Oct 30 21:09:04.688718 ip-10-0-142-232 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:09:05.083155 ip-10-0-142-232 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:09:05.083155 ip-10-0-142-232 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:09:05.083155 ip-10-0-142-232 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:09:05.083155 ip-10-0-142-232 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:09:05.086602 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086510    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.086602 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086602    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.086602 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086606    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086609    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086612    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086615    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086618    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086621    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086624    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086627    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086629    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086634    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086638    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086641    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086644    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086646    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086649    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086652    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086654    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086657    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086665    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.086702 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086668    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086672    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086674    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086677    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086680    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086682    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086685    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086688    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086690    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086693    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086695    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086698    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086700    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086703    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086706    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086709    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086712    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086715    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086718    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.087151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086720    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086723    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086725    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086729    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086731    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086734    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086736    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086738    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086741    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086743    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086746    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086748    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086751    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086753    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086756    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086760    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086763    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086766    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086768    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086771    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.087623 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086773    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086776    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086778    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086781    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086783    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086786    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086788    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086791    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086795    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086798    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086800    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086803    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086806    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086808    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086811    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086813    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086816    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086819    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086822    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086824    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.088093 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086827    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086830    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086833    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086835    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086838    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086841    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086844    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086846    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086849    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086852    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086854    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086857    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086860    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086862    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086865    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086868    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086871    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086874    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086876    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086879    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.088615 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086882    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086884    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086887    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086892    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086895    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.086899    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087730    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087738    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087741    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087744    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087748    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087751    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087754    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087757    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087759    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087765    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087769    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087772    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.089178 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087775    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087778    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087781    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087783    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087786    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087789    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087792    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087794    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087797    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087799    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087803    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087805    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087808    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087810    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087813    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087816    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087818    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087821    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087826    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.089639 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087829    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087832    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087835    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087838    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087841    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087845    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087848    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087851    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087853    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087856    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087859    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087862    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087865    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087868    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087870    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087873    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087875    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087878    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087881    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087883    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.090103 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087886    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087889    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087891    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087894    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087897    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087899    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087902    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087904    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087907    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087910    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087912    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087915    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087918    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087920    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087923    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087925    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087928    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087934    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087937    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087939    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.090605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087942    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087944    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087947    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087949    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087952    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087955    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087958    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087960    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087963    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087966    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087968    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087971    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087973    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087976    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087978    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087981    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087983    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087986    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087988    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087991    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.091090 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087994    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087997    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.087999    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088002    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088004    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088007    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088010    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088012    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088015    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088021    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088024    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088027    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088030    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088032    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088035    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088566    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088572    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088575    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088577    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.091590 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088580    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088583    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088586    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088588    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088591    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088594    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088597    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088600    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088602    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088605    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088608    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088611    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088613    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088616    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088620    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088623    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088625    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088629    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088631    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.092044 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088634    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088637    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088639    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088645    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088648    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088651    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088654    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088656    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088660    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088662    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088665    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088667    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088670    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088672    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088675    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088677    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088680    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088682    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088685    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088688    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.092517 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088690    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088693    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088695    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088698    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088700    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088703    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088707    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088710    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088713    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088715    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088718    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088720    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088723    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088726    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088729    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088735    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088738    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088741    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088744    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.092999 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088747    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088750    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088752    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088755    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088757    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088760    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088762    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088765    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088767    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088770    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088772    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088775    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088778    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088780    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088783    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088785    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088788    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088791    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088794    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088797    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.093496 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088800    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088802    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088805    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088807    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088810    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088812    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088814    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088817    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088821    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088824    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088826    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088829    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088837    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088839    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088842    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088845    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088847    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088850    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088852    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088855    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.093964 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088857    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088860    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088865    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.088869    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089955    2568 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089966    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089973    2568 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089978    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089984    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089987    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089992    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.089997    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090001    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090004    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090007    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090011    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090014    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090019    2568 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090022    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090025    2568 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090030    2568 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090033    2568 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090039    2568 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:09:05.094446 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090042    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090045    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090049    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090053    2568 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090057    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090060    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090063    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090067    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090070    2568 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090073    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090076    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090080    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090083    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090088    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090091    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090094    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090097    2568 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090100    2568 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090103    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090108    2568 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090111    2568 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090114    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090118    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090121    2568 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:09:05.094990 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090126    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090129    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090132    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090135    2568 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090138    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090142    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090145    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090148    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090152    2568 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090154    2568 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090157    2568 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090165    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090168    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090172    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090175    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090178    2568 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090181    2568 flags.go:64] FLAG: --help="false"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090184    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090187    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090191    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090193    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090197    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090201    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090203    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:09:05.095620 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090207    2568 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090209    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090212    2568 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090215    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090218    2568 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090221    2568 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090225    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090227    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090230    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090233    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090236    2568 flags.go:64] FLAG: --lock-file=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090239    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090242    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090245    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090251    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090254    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090257    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090260    2568 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090263    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090267    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090271    2568 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090274    2568 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090279    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090283    2568 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090287    2568 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:09:05.096257 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090290    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090313    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090316    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090320    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090323    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090326    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090330    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090337    2568 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090341    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090344    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090347    2568 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090360    2568 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090363    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090366    2568 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090370    2568 flags.go:64] FLAG: --port="10250"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090373    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090376    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00121f904ebad0248"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090380    2568 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090383    2568 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090386    2568 flags.go:64] FLAG: --register-node="true"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090389    2568 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090393    2568 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090396    2568 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090399    2568 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090402    2568 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090407    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:09:05.096933 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090410    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090415    2568 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090418    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090421    2568 flags.go:64] FLAG: --runonce="false"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090425    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090428    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090431    2568 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090434    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090436    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090440    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090443    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090446    2568 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090449    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090452    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090455    2568 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090458    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090461    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090464    2568 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090467    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090473    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090476    2568 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090479    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090484    2568 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090487    2568 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090490    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:09:05.097565 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090493    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090496    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090499    2568 flags.go:64] FLAG: --v="2"
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090504    2568 flags.go:64] FLAG: --version="false"
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090508    2568 flags.go:64] FLAG: --vmodule=""
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090512    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090516    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090675    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090681    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090684    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090687    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090690    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090693    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090696    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090698    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090701    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090704    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090706    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090709    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090712    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090714    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090716    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.098151 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090719    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090722    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090726    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090728    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090731    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090734    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090736    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090739    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090742    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090744    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090747    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090749    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090752    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090754    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090757    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090760    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090762    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090765    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090769    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090771    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.098700 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090774    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090777    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090780    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090782    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090785    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090787    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090790    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090792    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090795    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090797    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090800    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090802    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090805    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090808    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090811    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090814    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090817    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090819    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090822    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090824    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.099221 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090827    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090829    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090831    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090834    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090837    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090840    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090842    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090845    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090847    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090850    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090853    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090856    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090858    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090861    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090864    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090867    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090869    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090872    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090874    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090877    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.099884 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090879    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090882    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090885    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090887    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090890    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090893    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090897    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090899    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090902    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090905    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090907    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090910    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090914    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090918    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090921    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090925    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090929    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090932    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.100594 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090935    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090937    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090940    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090942    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090947    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090949    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090952    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090955    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090957    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090960    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090963    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090965    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.090968    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.090996    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091035    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091040    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.101251 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091043    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091045    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091048    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091050    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091054    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091058    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091061    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091063    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091066    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091069    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091071    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091074    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091077    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091080    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091084    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091086    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091090    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091094    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.101729 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091098    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091100    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091105    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091107    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091110    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091113    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091120    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091122    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091125    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091128    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091131    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091133    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091136    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091138    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091141    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091143    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091146    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091149    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091151    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.102158 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091155    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091158    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091161    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091164    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091166    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091169    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091172    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091174    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091177    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091180    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091182    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091184    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091187    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091190    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091192    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091196    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091199    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091201    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091204    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091206    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.102638 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091208    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091211    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091214    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091217    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091219    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091222    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091224    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091227    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091230    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091232    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091235    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091237    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091241    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091243    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091246    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091249    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091251    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091254    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091256    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091259    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.103140 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091261    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091264    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091267    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091269    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091272    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091274    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091277    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091281    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091283    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091286    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091288    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091291    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091293    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091296    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091298    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091301    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091304    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091307    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091309    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091312    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.103635 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091314    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091317    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091319    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091322    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091326    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091330    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.091332    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.098504    2568 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.098524    2568 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.098542    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.098547    2568 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098614    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098619    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098623    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098626    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098629    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098631    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098634    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098637    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.104166 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098639    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098642    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098645    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098648    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098650    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098653    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098655    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098658    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098660    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098663    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098666    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098669    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098671    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098674    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098677    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098679    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098682    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098685    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098688    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098691    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.104651 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098693    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098704    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098709    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098713    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098717    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098720    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098722    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098725    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098727    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098731    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098733    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098736    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098739    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098741    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098744    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098747    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098749    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098752    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098754    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.105144 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098757    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098760    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098762    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098765    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098767    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098769    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098774    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098777    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098781    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098785    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098787    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098790    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098793    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098796    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098798    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098807    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098810    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098812    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098815    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098817    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.105605 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098820    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098823    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098825    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098828    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098830    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098833    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098836    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098838    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098841    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098843    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098846    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098848    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098851    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098853    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098856    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098858    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098861    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098864    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098866    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098869    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.106115 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098871    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098874    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098877    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098879    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098882    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098884    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098887    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098889    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098898    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098901    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098903    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098906    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098908    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098911    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098913    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098916    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098919    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098921    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.106614 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.098924    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.098957    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101052    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101064    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101068    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101071    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101074    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101078    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101080    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101084    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101086    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101089    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101092    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101094    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101098    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101100    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.107078 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101103    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101106    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101109    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101112    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101115    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101117    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101120    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101123    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101126    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101128    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101131    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101134    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101136    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101140    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101143    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101146    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101150    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101152    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101155    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.107486 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101158    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101160    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101163    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101166    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101168    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101171    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101174    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101176    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101179    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101182    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101185    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101187    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101190    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101192    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101195    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101198    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101202    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101208    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101212    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.107957 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101216    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101219    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101222    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101227    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101229    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101232    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101235    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101237    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101240    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101243    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101245    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101248    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101251    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101254    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101257    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101259    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101262    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101265    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101268    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101271    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.108432 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101273    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101276    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101278    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101281    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101284    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101286    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101289    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101292    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101294    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101297    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101299    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101303    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101306    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101309    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101312    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101314    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101317    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101319    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101322    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.108953 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101325    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101327    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101330    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101332    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101335    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101337    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101340    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101342    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101345    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101347    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101369    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101372    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101375    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101378    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101380    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.101415    2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:09:05.109410 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101459    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101465    2568 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101468    2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101472    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101474    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101477    2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101480    2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101482    2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101485    2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101488    2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101491    2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101494    2568 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101497    2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101499    2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101502    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101505    2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101507    2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101510    2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101513    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101515    2568 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:05.109806 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101518    2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101521    2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101523    2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101526    2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101528    2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101531    2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101534    2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101536    2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101539    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101542    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101544    2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101547    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101549    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101552    2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101554    2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101557    2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101560    2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101563    2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101566    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101568    2568 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:05.110273 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101571    2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101573    2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101576    2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101579    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101581    2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101584    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101587    2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101590    2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101592    2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101595    2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101597    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101600    2568 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101603    2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101605    2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101608    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101610    2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101613    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101615    2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101618    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:05.110867 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101620    2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101623    2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101626    2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101628    2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101631    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101633    2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101636    2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101638    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101641    2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101644    2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101646    2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101649    2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101651    2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101654    2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101656    2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101661    2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101665    2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101667    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101670    2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101672    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:05.111382 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101676    2568 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101680    2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101686    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101690    2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101695    2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101698    2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101701    2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101703    2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101706    2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101709    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101712    2568 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101714    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101717    2568 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101719    2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101722    2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101725    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101728    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101731    2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101734    2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:05.111904 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101737    2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101739    2568 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101742    2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101745    2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101748    2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101752    2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101755    2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:05.101757    2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.101818    2568 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.103758    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.104851    2568 server.go:1014] "Starting client certificate rotation"
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.104951    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.105712    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.106836    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:09:05.112348 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.109975    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:09:05.121005 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.120985    2568 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:09:05.124645 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.124626    2568 log.go:25] "Validated CRI v1 image API"
Oct 30 21:09:05.125572 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.125537    2568 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:09:05.131364 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.131333    2568 fs.go:135] Filesystem UUIDs: map[52f30e7f-b0cf-465b-9b0c-f4ee43bf6849:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c92ad68e-9432-4cb3-a3e2-9d2e0ba954db:/dev/nvme0n1p3]
Oct 30 21:09:05.131432 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.131369    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:09:05.137510 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.137342    2568 manager.go:217] Machine: {Timestamp:2026-10-30 21:09:05.135894861 +0000 UTC m=+0.334890997 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100648 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20a09fc4cf9a43a1abb76d4471cd06 SystemUUID:ec20a09f-c4cf-9a43-a1ab-b76d4471cd06 BootID:610b313d-715e-4f6f-9f3a-f2c9e77b6bcf Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:66:46:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:66:46:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:4f:99:40:2a:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:09:05.137510 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.137509    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:09:05.137622 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.137606    2568 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:09:05.140163 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.140137    2568 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:09:05.140326 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.140172    2568 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:09:05.140392 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.140329    2568 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:09:05.140392 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.140336    2568 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:09:05.140392 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.140387    2568 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:09:05.140477 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.140409    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:09:05.141085 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.141074    2568 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:09:05.144026 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144013    2568 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:09:05.144180 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144171    2568 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:09:05.144741 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144731    2568 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:09:05.144773 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144750    2568 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:09:05.144773 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144761    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:09:05.144773 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144770    2568 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:09:05.144860 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.144779    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:09:05.145298 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.145284    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:09:05.148540 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.148458    2568 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:09:05.153599 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.153583    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:09:05.154527 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154513    2568 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:09:05.154574 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154546    2568 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:09:05.154734 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154722    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:09:05.154763 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154740    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:09:05.154763 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154747    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:09:05.154763 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154761    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:09:05.154834 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154768    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:09:05.154834 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.154778    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:09:05.155831 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.155821    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:09:05.155875 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.155831    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:09:05.155875 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.155839    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:09:05.155875 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.155845    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:09:05.155875 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.155854    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:09:05.155875 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.155863    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:09:05.156772 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.156763    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:09:05.156806 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.156776    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:09:05.160680 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.160664    2568 server.go:1263] "Started kubelet"
Oct 30 21:09:05.160815 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.160756    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:09:05.160937 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.160835    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:09:05.160937 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.160738    2568 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:09:05.161539 ip-10-0-142-232 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:09:05.161822 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.161804    2568 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:09:05.163453 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.163435    2568 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:09:05.166508 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.166490    2568 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:09:05.168051 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.168034    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:09:05.168456 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.168437    2568 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:09:05.168785 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.168765    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.169182 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169164    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:09:05.169182 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169165    2568 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:09:05.169312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169193    2568 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:09:05.169312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169296    2568 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:09:05.169312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169304    2568 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:09:05.169427 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169343    2568 factory.go:55] Registering systemd factory
Oct 30 21:09:05.169427 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169373    2568 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:09:05.169574 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169558    2568 factory.go:153] Registering CRI-O factory
Oct 30 21:09:05.169574 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169574    2568 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:09:05.169665 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169630    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:09:05.169665 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169646    2568 factory.go:103] Registering Raw factory
Oct 30 21:09:05.169665 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.169659    2568 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:09:05.170710 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.170693    2568 manager.go:319] Starting recovery of all containers
Oct 30 21:09:05.183667 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.183645    2568 manager.go:324] Recovery completed
Oct 30 21:09:05.185077 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.185054    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:09:05.185338 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.185306    2568 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:09:05.186055 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.185067    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-232.ec2.internal.18e36ad6466c62be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-232.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-232.ec2.internal,},FirstTimestamp:2026-10-30 21:09:05.160643262 +0000 UTC m=+0.359639399,LastTimestamp:2026-10-30 21:09:05.160643262 +0000 UTC m=+0.359639399,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-232.ec2.internal,}"
Oct 30 21:09:05.188400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.188385    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.190736 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.190716    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.190804 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.190755    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.190804 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.190767    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.191391 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.191373    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.191498 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.191347    2568 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:09:05.191498 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.191420    2568 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:09:05.191498 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.191439    2568 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:09:05.192152 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.192134    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.192210 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.192170    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.192210 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.192184    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.192713 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.192700    2568 policy_none.go:50] "Start"
Oct 30 21:09:05.192763 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.192718    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:09:05.192763 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.192733    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:09:05.193726 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.193711    2568 policy_none.go:44] "Start"
Oct 30 21:09:05.194111 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.194093    2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:09:05.194210 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.194147    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5g2n"
Oct 30 21:09:05.201838 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.201808    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5g2n"
Oct 30 21:09:05.203657 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.203620    2568 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:09:05.203797 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.203650    2568 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:09:05.203884 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.203856    2568 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:09:05.218972 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.218876    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-232.ec2.internal.18e36ad648379233  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-232.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-232.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-232.ec2.internal,},FirstTimestamp:2026-10-30 21:09:05.190736435 +0000 UTC m=+0.389732575,LastTimestamp:2026-10-30 21:09:05.190736435 +0000 UTC m=+0.389732575,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-232.ec2.internal,}"
Oct 30 21:09:05.224042 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.224006    2568 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:09:05.226650 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.226638    2568 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:09:05.226695 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.226669    2568 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:09:05.226695 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.226678    2568 server.go:87] "Starting device plugin registration server"
Oct 30 21:09:05.227009 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.226995    2568 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:09:05.227088 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.227013    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:09:05.227192 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.227178    2568 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:09:05.227267 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.227256    2568 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:09:05.227330 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.227266    2568 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:09:05.227839 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.227821    2568 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:09:05.227917 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.227874    2568 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.298775 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.298734    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:09:05.299986 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.299932    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:09:05.299986 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.299954    2568 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:09:05.299986 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.299980    2568 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:09:05.300158 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.300021    2568 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:09:05.314870 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.314692    2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:05.327711 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.327688    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.328730 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.328702    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.328838 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.328743    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.328838 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.328755    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.328838 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.328783    2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.347658 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.347590    2568 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.347658 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.347624    2568 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-232.ec2.internal\": node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.392219 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.392192    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.400568 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.400541    2568 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"]
Oct 30 21:09:05.400677 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.400616    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.401618 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.401603    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.401698 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.401631    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.401698 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.401648    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.403158 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403145    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.403321 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403306    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.403379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403340    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.403956 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403941    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.404024 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403972    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.404024 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403983    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.404093 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.403941    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.404093 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.404041    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.404093 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.404053    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.405800 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.405780    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.405890 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.405808    2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:09:05.406601 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.406583    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:09:05.406700 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.406613    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:09:05.406700 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.406625    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:09:05.427775 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.427744    2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-232.ec2.internal\" not found" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.431463 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.431441    2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-232.ec2.internal\" not found" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.471617 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.471591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.471716 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.471620    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.471716 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.471641    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d772f23118a807c4f6e4705c5121a0bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" (UID: \"d772f23118a807c4f6e4705c5121a0bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.492451 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.492425    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.572541 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.572508    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.572613 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.572546    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.572613 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.572562    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d772f23118a807c4f6e4705c5121a0bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" (UID: \"d772f23118a807c4f6e4705c5121a0bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.572677 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.572625    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d772f23118a807c4f6e4705c5121a0bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" (UID: \"d772f23118a807c4f6e4705c5121a0bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.572677 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.572627    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.572677 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.572627    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.592865 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.592825    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.693003 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.692939    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.731185 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.731156    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.734867 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:05.734831    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:05.793591 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.793551    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.894097 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.894050    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:05.994626 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:05.994538    2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:09:07.321758 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.321724    2568 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:07.322672 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.322654    2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:07.346231 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.346207    2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:09:07.348084 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.348068    2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:07.358471 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.358449    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:09:07.358643 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.358597    2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:09:07.358695 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.358628    2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:09:07.358695 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.358659    2568 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a684b1ccd511c4728b56d83348d69821-be62a566440ea101.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.142.232:51004->3.21.20.129:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:09:07.398750 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.398724    2568 apiserver.go:52] "Watching apiserver"
Oct 30 21:09:07.421876 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.421845    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:09:07.458631 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.458571    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:04:06 +0000 UTC" deadline="2028-07-17 23:20:38.612315254 +0000 UTC"
Oct 30 21:09:07.458631 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.458627    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15026h11m31.15369186s"
Oct 30 21:09:07.462492 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.462443    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd772f23118a807c4f6e4705c5121a0bf.slice/crio-4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d WatchSource:0}: Error finding container 4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d: Status 404 returned error can't find the container with id 4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d
Oct 30 21:09:07.462725 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.462708    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf70e9740ecdd5b859325e61fd6d5f34.slice/crio-32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b WatchSource:0}: Error finding container 32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b: Status 404 returned error can't find the container with id 32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b
Oct 30 21:09:07.465965 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.465945    2568 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:09:07.466231 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.466217    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-tn775","openshift-image-registry/node-ca-8v7dh","openshift-network-operator/iptables-alerter-nb5hx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw","kube-system/global-pull-secret-syncer-8w46l","openshift-dns/node-resolver-9bv86","openshift-multus/network-metrics-daemon-fbzn9","openshift-network-diagnostics/network-check-target-pb8b2","kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal","openshift-cluster-node-tuning-operator/tuned-svkg2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal","openshift-multus/multus-additional-cni-plugins-k57t9","openshift-multus/multus-sm2w6","openshift-ovn-kubernetes/ovnkube-node-x52r2"]
Oct 30 21:09:07.468263 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.468243    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:07.468371 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.468344    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:07.469377 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.469288    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.469538 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.469514    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:09:07.470626 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.470595    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.471730 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.471708    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.473072 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.473054    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.473140 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.473126    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:07.474048 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.474032    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.475543 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.475524    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.476644 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.476627    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.477943 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.477921    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.479132 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.479114    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.480226 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.480210    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:07.480301 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.480264    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:07.481726 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.481707    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.483797 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.483782    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.490291 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.490273    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5899\""
Oct 30 21:09:07.490291 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.490290    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d6fgf\""
Oct 30 21:09:07.490453 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.490376    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:09:07.490453 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.490427    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hnk74\""
Oct 30 21:09:07.490453 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.490447    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nxx4w\""
Oct 30 21:09:07.490554 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.490515    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jv27h\""
Oct 30 21:09:07.494459 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.494443    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:09:07.494587 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.494572    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:09:07.494670 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.494658    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjvtw\""
Oct 30 21:09:07.494793 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.494777    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t284x\""
Oct 30 21:09:07.494893 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.494824    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2tg5\""
Oct 30 21:09:07.495052 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.495030    2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:09:07.495134 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.495099    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6266f"
Oct 30 21:09:07.495711 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.495691    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8m2s\""
Oct 30 21:09:07.524346 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.524325    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:09:07.529334 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-sys\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.529449 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529344    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.529449 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529367    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-cnibin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.529449 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529388    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-systemd-units\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.529563 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-var-lib-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.529563 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529484    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88e4b3aa-5e27-4609-ae83-e74875a7e8da-iptables-alerter-script\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.529563 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-slash\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.529563 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529541    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.529735 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-modprobe-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.529735 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529677    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-netd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.529735 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529701    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:07.529735 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cnibin\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.529880 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529765    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.529880 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529787    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-etc-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.529880 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529803    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-log-socket\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.529880 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530064 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529893    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-script-lib\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530064 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529926    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-device-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.530064 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529953    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-kubelet-config\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.530064 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.529994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-system-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530064 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530035    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-bin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530064 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530062    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-conf-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-run\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530113    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88e4b3aa-5e27-4609-ae83-e74875a7e8da-host-slash\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530141    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-var-lib-kubelet\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530182    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkxm\" (UniqueName: \"kubernetes.io/projected/19448ad0-e5a8-4760-b4c5-e8410e14af1b-kube-api-access-2nkxm\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxvk\" (UniqueName: \"kubernetes.io/projected/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kube-api-access-khxvk\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-kubelet\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530278    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-ovn\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530296    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-config\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530320    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-conf\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530351    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-systemd\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530389    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgz58\" (UniqueName: \"kubernetes.io/projected/a2806653-74fd-4d37-a4ba-459232f41e74-kube-api-access-jgz58\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530406    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-multus-certs\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530427    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530443    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-node-log\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530481    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ss9\" (UniqueName: \"kubernetes.io/projected/67203193-932d-445f-b14d-457040dcee8e-kube-api-access-c4ss9\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530510    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530536    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b394b4-d048-4139-a5eb-c32918e048d2-hosts-file\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.530588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530574    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-multus\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530599    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-multus-daemon-config\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-bin\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274cp\" (UniqueName: \"kubernetes.io/projected/054fd709-2d73-4ce0-ab94-0df9fbd1921a-kube-api-access-274cp\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530671    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c567c895-2164-4fdc-a4b4-4df9af394efc-agent-certs\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530689    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-dbus\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530731    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-socket-dir-parent\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530776    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-netns\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530805    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c567c895-2164-4fdc-a4b4-4df9af394efc-konnectivity-ca\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530829    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530860    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530883    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhqr\" (UniqueName: \"kubernetes.io/projected/88e4b3aa-5e27-4609-ae83-e74875a7e8da-kube-api-access-cwhqr\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530914    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxwn\" (UniqueName: \"kubernetes.io/projected/93b394b4-d048-4139-a5eb-c32918e048d2-kube-api-access-rcxwn\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530934    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysconfig\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.530982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-cni-binary-copy\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.530994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-systemd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531025    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67203193-932d-445f-b14d-457040dcee8e-serviceca\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531063    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-etc-tuned\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531089    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-hostroot\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531133    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5f5\" (UniqueName: \"kubernetes.io/projected/900d3f13-ab1c-478e-9851-a70264d92012-kube-api-access-cs5f5\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531159    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-socket-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b394b4-d048-4139-a5eb-c32918e048d2-tmp-dir\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531228    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-host\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531260    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531292    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531325    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcdb\" (UniqueName: \"kubernetes.io/projected/9a33b283-c453-4742-9142-173f02cadeab-kube-api-access-4fcdb\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531344    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-lib-modules\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531368    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-registration-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531405    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531433    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-sys-fs\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.531549 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531461    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovn-node-metrics-cert\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531487    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-system-cni-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531509    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-kubernetes\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531531    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-tmp\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-netns\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531581    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-k8s-cni-cncf-io\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-etc-kubernetes\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531651    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-kubelet\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531690    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-env-overrides\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.532113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.531720    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67203193-932d-445f-b14d-457040dcee8e-host\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.556260 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.556178    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal" event={"ID":"d772f23118a807c4f6e4705c5121a0bf","Type":"ContainerStarted","Data":"4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d"}
Oct 30 21:09:07.557236 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.557214    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" event={"ID":"cf70e9740ecdd5b859325e61fd6d5f34","Type":"ContainerStarted","Data":"32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b"}
Oct 30 21:09:07.583976 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.583916    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.584214 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584189    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:09:07.584298 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584221    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.584298 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584261    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.584298 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584272    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584311    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584314    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584326    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584357    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584368    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584380    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584411    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584410    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584453    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.584463 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584464    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:09:07.584775 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584471    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.584775 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584467    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:09:07.584775 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.584467    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:09:07.588856 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.588836    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.588856 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.588855    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:09:07.589025 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.588862    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:09:07.589025 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.588874    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:09:07.589025 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.588924    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:09:07.632147 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632109    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.632147 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632145    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-etc-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632162    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-log-socket\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632189    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632231    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632237    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-etc-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632247    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632245    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-log-socket\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632261    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-script-lib\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632299    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-device-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632319    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-kubelet-config\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632335    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-system-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632350    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-bin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.632379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632377    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-device-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632382    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-conf-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632416    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-kubelet-config\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-run\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632416    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-bin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632441    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-system-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632454    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-conf-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632451    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88e4b3aa-5e27-4609-ae83-e74875a7e8da-host-slash\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-var-lib-kubelet\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632508    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88e4b3aa-5e27-4609-ae83-e74875a7e8da-host-slash\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632528    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632543    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-run\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632553    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkxm\" (UniqueName: \"kubernetes.io/projected/19448ad0-e5a8-4760-b4c5-e8410e14af1b-kube-api-access-2nkxm\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.632554    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-var-lib-kubelet\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633015 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633011    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-script-lib\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633065    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxvk\" (UniqueName: \"kubernetes.io/projected/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kube-api-access-khxvk\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633107    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633141    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-kubelet\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-ovn\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633190    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-config\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633221    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-kubelet\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633266    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-ovn\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633301    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633307    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-conf\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633361    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-systemd\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633387    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgz58\" (UniqueName: \"kubernetes.io/projected/a2806653-74fd-4d37-a4ba-459232f41e74-kube-api-access-jgz58\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633414    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-systemd\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633413    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-multus-certs\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633413    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633444    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633456    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-multus-certs\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.633691 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633445    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-conf\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633470    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-node-log\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633481    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ss9\" (UniqueName: \"kubernetes.io/projected/67203193-932d-445f-b14d-457040dcee8e-kube-api-access-c4ss9\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633502    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-node-log\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633522    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633554    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b394b4-d048-4139-a5eb-c32918e048d2-hosts-file\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633582    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-multus\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633625    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-multus-daemon-config\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633627    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b394b4-d048-4139-a5eb-c32918e048d2-hosts-file\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633647    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-multus\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633676    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-bin\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.633714    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633721    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-bin\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.633762    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:08.133747637 +0000 UTC m=+2.079493522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633724    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274cp\" (UniqueName: \"kubernetes.io/projected/054fd709-2d73-4ce0-ab94-0df9fbd1921a-kube-api-access-274cp\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633711    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-config\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.634513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633817    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c567c895-2164-4fdc-a4b4-4df9af394efc-agent-certs\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633845    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-dbus\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-socket-dir-parent\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633899    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-netns\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633928    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c567c895-2164-4fdc-a4b4-4df9af394efc-konnectivity-ca\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633952    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.633992    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-socket-dir-parent\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhqr\" (UniqueName: \"kubernetes.io/projected/88e4b3aa-5e27-4609-ae83-e74875a7e8da-kube-api-access-cwhqr\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634048    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-dbus\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634054    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxwn\" (UniqueName: \"kubernetes.io/projected/93b394b4-d048-4139-a5eb-c32918e048d2-kube-api-access-rcxwn\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysconfig\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634095    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-netns\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634115    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-cni-binary-copy\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-systemd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634146    2568 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67203193-932d-445f-b14d-457040dcee8e-serviceca\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634169    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-multus-daemon-config\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.635312 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-etc-tuned\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634212    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-hostroot\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634226    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-systemd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634227    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysconfig\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634237    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5f5\" (UniqueName: \"kubernetes.io/projected/900d3f13-ab1c-478e-9851-a70264d92012-kube-api-access-cs5f5\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-socket-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634287    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b394b4-d048-4139-a5eb-c32918e048d2-tmp-dir\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634312    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-host\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634328    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634337    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634365    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634389    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcdb\" (UniqueName: \"kubernetes.io/projected/9a33b283-c453-4742-9142-173f02cadeab-kube-api-access-4fcdb\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634441    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-lib-modules\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634468    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-registration-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634491    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-hostroot\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634492    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-sys-fs\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovn-node-metrics-cert\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634620    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-socket-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634622    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-system-cni-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634647    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634659    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-system-cni-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634674    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-kubernetes\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634709    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-sys-fs\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.634716    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634725    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-tmp\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634751    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-netns\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634756    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-registration-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.634783    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:08.134756998 +0000 UTC m=+2.080502914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634793    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-netns\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634828    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-lib-modules\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634879    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-kubernetes\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634945    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.634697    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-cni-binary-copy\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.636940 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635001    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635048    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-host\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635104    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-k8s-cni-cncf-io\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635128    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-etc-kubernetes\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635148    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-kubelet\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635169    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-env-overrides\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635188    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67203193-932d-445f-b14d-457040dcee8e-host\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635205    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-etc-kubernetes\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635238    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-kubelet\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635274    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67203193-932d-445f-b14d-457040dcee8e-host\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635308    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-k8s-cni-cncf-io\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635209    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-sys\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635244    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-sys\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635343    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635377    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-cnibin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635401    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-systemd-units\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-var-lib-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88e4b3aa-5e27-4609-ae83-e74875a7e8da-iptables-alerter-script\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.637738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635477    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-slash\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635501    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635476    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c567c895-2164-4fdc-a4b4-4df9af394efc-konnectivity-ca\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635524    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-modprobe-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635560    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-netd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635589    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635616    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-modprobe-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635635    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cnibin\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635669    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635705    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-env-overrides\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635714    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cnibin\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635710    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-var-lib-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635746    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-slash\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635787    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-cnibin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635792    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-netd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.635834    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-systemd-units\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.636176    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67203193-932d-445f-b14d-457040dcee8e-serviceca\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.638213 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.636266    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88e4b3aa-5e27-4609-ae83-e74875a7e8da-iptables-alerter-script\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.638686 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.636299    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.638686 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.636399    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b394b4-d048-4139-a5eb-c32918e048d2-tmp-dir\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.638686 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.637972    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-etc-tuned\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.638686 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.638017    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovn-node-metrics-cert\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.638686 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.638136    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c567c895-2164-4fdc-a4b4-4df9af394efc-agent-certs\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.638686 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.638213    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-tmp\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.653068 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.653039    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgz58\" (UniqueName: \"kubernetes.io/projected/a2806653-74fd-4d37-a4ba-459232f41e74-kube-api-access-jgz58\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.653712 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.653518    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxvk\" (UniqueName: \"kubernetes.io/projected/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kube-api-access-khxvk\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.653712 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.653560    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkxm\" (UniqueName: \"kubernetes.io/projected/19448ad0-e5a8-4760-b4c5-e8410e14af1b-kube-api-access-2nkxm\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.653712 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.653579    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274cp\" (UniqueName: \"kubernetes.io/projected/054fd709-2d73-4ce0-ab94-0df9fbd1921a-kube-api-access-274cp\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.656273 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.656249    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhqr\" (UniqueName: \"kubernetes.io/projected/88e4b3aa-5e27-4609-ae83-e74875a7e8da-kube-api-access-cwhqr\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.656618 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.656585    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxwn\" (UniqueName: \"kubernetes.io/projected/93b394b4-d048-4139-a5eb-c32918e048d2-kube-api-access-rcxwn\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.656715 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.656626    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcdb\" (UniqueName: \"kubernetes.io/projected/9a33b283-c453-4742-9142-173f02cadeab-kube-api-access-4fcdb\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.656782 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.656761    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ss9\" (UniqueName: \"kubernetes.io/projected/67203193-932d-445f-b14d-457040dcee8e-kube-api-access-c4ss9\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.657320 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.657305    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:07.657354 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.657323    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:07.657354 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.657333    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:07.657411 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:07.657387    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:08.157371157 +0000 UTC m=+2.103117055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:07.659327 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.659307    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5f5\" (UniqueName: \"kubernetes.io/projected/900d3f13-ab1c-478e-9851-a70264d92012-kube-api-access-cs5f5\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:07.706341 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.706312    2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:07.781041 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.781003    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sm2w6"
Oct 30 21:09:07.787998 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.787977    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:09:07.789804 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.789769    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a33b283_c453_4742_9142_173f02cadeab.slice/crio-7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a WatchSource:0}: Error finding container 7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a: Status 404 returned error can't find the container with id 7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a
Oct 30 21:09:07.795471 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.795447    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:09:07.795620 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.795565    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e4b3aa_5e27_4609_ae83_e74875a7e8da.slice/crio-ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4 WatchSource:0}: Error finding container ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4: Status 404 returned error can't find the container with id ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4
Oct 30 21:09:07.801161 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.801142    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:09:07.802617 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.802583    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf6607a_ab80_47b6_ad1c_e33939b8dc1e.slice/crio-4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99 WatchSource:0}: Error finding container 4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99: Status 404 returned error can't find the container with id 4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99
Oct 30 21:09:07.809872 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.809850    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b394b4_d048_4139_a5eb_c32918e048d2.slice/crio-ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b WatchSource:0}: Error finding container ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b: Status 404 returned error can't find the container with id ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b
Oct 30 21:09:07.813364 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.813348    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:07.814414 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.814397    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:07.818688 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.818669    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:09:07.825801 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.824980    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:09:07.827264 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.827241    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054fd709_2d73_4ce0_ab94_0df9fbd1921a.slice/crio-a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92 WatchSource:0}: Error finding container a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92: Status 404 returned error can't find the container with id a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92
Oct 30 21:09:07.831362 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.831333    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67203193_932d_445f_b14d_457040dcee8e.slice/crio-c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2 WatchSource:0}: Error finding container c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2: Status 404 returned error can't find the container with id c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2
Oct 30 21:09:07.834517 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:07.834501    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:09:07.835891 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.835865    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2806653_74fd_4d37_a4ba_459232f41e74.slice/crio-9fc5326469a97221ce8108dbcb1eba64c90d0d2443e7c272c43c7184a6de636a WatchSource:0}: Error finding container 9fc5326469a97221ce8108dbcb1eba64c90d0d2443e7c272c43c7184a6de636a: Status 404 returned error can't find the container with id 9fc5326469a97221ce8108dbcb1eba64c90d0d2443e7c272c43c7184a6de636a
Oct 30 21:09:07.843113 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:07.843082    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19448ad0_e5a8_4760_b4c5_e8410e14af1b.slice/crio-63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2 WatchSource:0}: Error finding container 63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2: Status 404 returned error can't find the container with id 63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2
Oct 30 21:09:08.001149 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.001109    2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:08.139926 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.139277    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:08.139926 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.139366    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:08.139926 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.139487    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:08.139926 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.139552    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.139533344 +0000 UTC m=+3.085279232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:08.140204 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.139927    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:08.140204 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.140011    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.139987017 +0000 UTC m=+3.085732918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:08.240712 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.239942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:08.240712 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.240163    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:08.240712 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.240201    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:08.240712 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.240216    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:08.240712 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:08.240278    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:09.240260325 +0000 UTC m=+3.186006213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:08.497104 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.496960    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:04:07 +0000 UTC" deadline="2028-06-05 18:16:51.168383464 +0000 UTC"
Oct 30 21:09:08.497104 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.497002    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14013h7m42.671386372s"
Oct 30 21:09:08.577629 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.577412    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92"}
Oct 30 21:09:08.581961 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.581858    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9bv86" event={"ID":"93b394b4-d048-4139-a5eb-c32918e048d2","Type":"ContainerStarted","Data":"ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b"}
Oct 30 21:09:08.585632 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.585582    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sm2w6" event={"ID":"9a33b283-c453-4742-9142-173f02cadeab","Type":"ContainerStarted","Data":"7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a"}
Oct 30 21:09:08.591853 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.591770    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-svkg2" event={"ID":"a2806653-74fd-4d37-a4ba-459232f41e74","Type":"ContainerStarted","Data":"9fc5326469a97221ce8108dbcb1eba64c90d0d2443e7c272c43c7184a6de636a"}
Oct 30 21:09:08.600076 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.599994    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8v7dh" event={"ID":"67203193-932d-445f-b14d-457040dcee8e","Type":"ContainerStarted","Data":"c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2"}
Oct 30 21:09:08.612581 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.612477    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw" event={"ID":"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e","Type":"ContainerStarted","Data":"4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99"}
Oct 30 21:09:08.626769 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.626665    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nb5hx" event={"ID":"88e4b3aa-5e27-4609-ae83-e74875a7e8da","Type":"ContainerStarted","Data":"ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4"}
Oct 30 21:09:08.636742 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.636595    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerStarted","Data":"63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2"}
Oct 30 21:09:08.641137 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.641050    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tn775" event={"ID":"c567c895-2164-4fdc-a4b4-4df9af394efc","Type":"ContainerStarted","Data":"4979cc278591ce7eded65e70bfdd43754320c03d4fbdad853dec669bb87b491a"}
Oct 30 21:09:08.768043 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:08.767963    2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:09.147422 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.147385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:09.147626 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.147450    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:09.147626 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.147585    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:09.147746 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.147678    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:11.147644631 +0000 UTC m=+5.093390534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:09.147809 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.147781    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:09.147862 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.147831    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:11.14781902 +0000 UTC m=+5.093564924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:09.229309 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.229279    2568 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:09.248729 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.248689    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:09.248981 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.248945    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:09.248981 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.248973    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:09.249100 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.248988    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:09.249100 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.249052    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:11.249033367 +0000 UTC m=+5.194779268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:09.497274 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.497146    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:04:07 +0000 UTC" deadline="2028-05-17 08:04:49.525679056 +0000 UTC"
Oct 30 21:09:09.497274 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.497213    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13546h55m40.028468687s"
Oct 30 21:09:09.553714 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.553499    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:09.553714 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.553637    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:09.554335 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.554045    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:09.554335 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.554190    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:09.554335 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:09.554243    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:09.554335 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:09.554296    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:11.166738 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.165449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:11.167157 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.166813    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:11.167157 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.165695    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:11.167157 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.167012    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:15.166994216 +0000 UTC m=+9.112740105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:11.167390 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.166949    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:11.167447 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.167435    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:15.167416733 +0000 UTC m=+9.113162639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:11.267308 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.267174    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:11.267464 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.267412    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:11.267464 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.267431    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:11.267464 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.267440    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:11.267636 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.267492    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:15.267479345 +0000 UTC m=+9.213225242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:11.554357 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.554319    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:11.554537 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.554450    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:11.554537 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.554504    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:11.554670 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.554564    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:11.554670 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.554641    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:11.554774 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:11.554713    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:11.662897 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.662855    2568 generic.go:356] "Generic (PLEG): container finished" podID="cf70e9740ecdd5b859325e61fd6d5f34" containerID="f73ddb974843f2b8547d06fa25901644f9bc939c371398b81e91494fcb60b08d" exitCode=0
Oct 30 21:09:11.663188 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.663150    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" event={"ID":"cf70e9740ecdd5b859325e61fd6d5f34","Type":"ContainerDied","Data":"f73ddb974843f2b8547d06fa25901644f9bc939c371398b81e91494fcb60b08d"}
Oct 30 21:09:11.668126 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.667952    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal" event={"ID":"d772f23118a807c4f6e4705c5121a0bf","Type":"ContainerStarted","Data":"ccdb38768ebfbab61c0269db83345e4eef09980a8f9c47eb70feb80d7e782a56"}
Oct 30 21:09:11.674555 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.674523    2568 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="8be45d996c7e3f461656858749790ad2c55abc52abfac64a3e6f5dea8e772ba2" exitCode=0
Oct 30 21:09:11.674666 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.674576    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerDied","Data":"8be45d996c7e3f461656858749790ad2c55abc52abfac64a3e6f5dea8e772ba2"}
Oct 30 21:09:11.736756 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:11.736689    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal" podStartSLOduration=4.736667983 podStartE2EDuration="4.736667983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:11.735745954 +0000 UTC m=+5.681491868" watchObservedRunningTime="2026-10-30 21:09:11.736667983 +0000 UTC m=+5.682413892"
Oct 30 21:09:12.682832 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:12.682794    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tn775" event={"ID":"c567c895-2164-4fdc-a4b4-4df9af394efc","Type":"ContainerStarted","Data":"de1369e49740e4d344096462ba9053dce8d140b6f347961e9ec995c6e344f120"}
Oct 30 21:09:12.687019 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:12.686991    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" event={"ID":"cf70e9740ecdd5b859325e61fd6d5f34","Type":"ContainerStarted","Data":"9e141d04f7f14e3e9c98ab973bc237b9916c8e5b878c8776af5e8a881ef75a88"}
Oct 30 21:09:12.722309 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:12.722236    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tn775" podStartSLOduration=2.504149778 podStartE2EDuration="6.722215791s" totalImagesPullingTime="4.218066013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:12.721031214 +0000 UTC m=+6.666777118" watchObservedRunningTime="2026-10-30 21:09:12.722215791 +0000 UTC m=+6.667961699"
Oct 30 21:09:12.759712 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:12.758802    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" podStartSLOduration=5.758781418 podStartE2EDuration="5.758781418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:12.758079728 +0000 UTC m=+6.703825637" watchObservedRunningTime="2026-10-30 21:09:12.758781418 +0000 UTC m=+6.704527543"
Oct 30 21:09:13.554930 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:13.554891    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:13.555137 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:13.555050    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:13.555462 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:13.555442    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:13.555553 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:13.555539    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:13.555633 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:13.555616    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:13.555756 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:13.555718    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:13.693754 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:13.693713    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8v7dh" event={"ID":"67203193-932d-445f-b14d-457040dcee8e","Type":"ContainerStarted","Data":"cb99f2f22bdf6125fbe4a8ff549869f1fd77f45769e63a5716720d32a9777011"}
Oct 30 21:09:14.701295 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:14.701205    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw" event={"ID":"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e","Type":"ContainerStarted","Data":"af8b1f48760edbda5e4d4f469e838f272579a9b27a37d8840966aa60d3b5d8e7"}
Oct 30 21:09:15.199540 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.199496    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:15.199783 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.199569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:15.199783 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.199722    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:15.199904 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.199791    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.199770867 +0000 UTC m=+17.145516755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:15.200344 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.200207    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:15.200344 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.200268    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.200253541 +0000 UTC m=+17.145999443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:15.300520 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.300462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:15.300708 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.300691    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:15.300849 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.300716    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:15.300849 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.300730    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:15.300849 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.300792    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:23.300773746 +0000 UTC m=+17.246519634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:15.396643 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.396574    2568 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:09:15.486887 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.486209    2568 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:09:15.396617407Z","UUID":"f9cd23bf-ae8b-47f1-8047-5262fee6ad64","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:09:15.488350 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.487992    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:09:15.488350 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.488025    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:09:15.553596 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.553527    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:15.553788 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.553665    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:15.554061 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.554021    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:15.554173 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.554119    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:15.554239 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.554181    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:15.554288 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:15.554255    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:15.708545 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:15.708508    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw" event={"ID":"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e","Type":"ContainerStarted","Data":"5265e7b6b3ad1ea9167dc112c2241b3cdd6b76c4a92a1a3814e19c4cee59dc90"}
Oct 30 21:09:16.648526 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.648284    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:16.648918 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.648893    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:16.687311 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.687211    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8v7dh" podStartSLOduration=5.553084282 podStartE2EDuration="10.687194154s" totalImagesPullingTime="5.134109872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:13.732416621 +0000 UTC m=+7.678162528" watchObservedRunningTime="2026-10-30 21:09:16.687194154 +0000 UTC m=+10.632940062"
Oct 30 21:09:16.713584 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.713547    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw" event={"ID":"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e","Type":"ContainerStarted","Data":"925867a4777c1ce525f2a3e18a281ca779d7c1471757b8a22954a23378883ff8"}
Oct 30 21:09:16.715963 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.715931    2568 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="5945605820d182f067de37b5bbb6af5ddca58ff0aef2968c3eff2a5a37d060fc" exitCode=0
Oct 30 21:09:16.716262 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.716204    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerDied","Data":"5945605820d182f067de37b5bbb6af5ddca58ff0aef2968c3eff2a5a37d060fc"}
Oct 30 21:09:16.717982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.717877    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9bv86" event={"ID":"93b394b4-d048-4139-a5eb-c32918e048d2","Type":"ContainerStarted","Data":"c7465243eb2d41402b89cbe27ea3a607642900d0a3f4a6245d2c18924400fa94"}
Oct 30 21:09:16.719551 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.719528    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sm2w6" event={"ID":"9a33b283-c453-4742-9142-173f02cadeab","Type":"ContainerStarted","Data":"9c607f8bb86f0848531e6d5f979522c7d7d270f6a00400a0d79c9e0891102754"}
Oct 30 21:09:16.755272 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.755216    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw" podStartSLOduration=2.496885663 podStartE2EDuration="10.75519728s" totalImagesPullingTime="8.258311617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:16.755129208 +0000 UTC m=+10.700875118" watchObservedRunningTime="2026-10-30 21:09:16.75519728 +0000 UTC m=+10.700943185"
Oct 30 21:09:16.844011 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:16.843558    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9bv86" podStartSLOduration=2.569152913 podStartE2EDuration="10.843540345s" totalImagesPullingTime="8.274387432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:16.79397021 +0000 UTC m=+10.739716124" watchObservedRunningTime="2026-10-30 21:09:16.843540345 +0000 UTC m=+10.789286254"
Oct 30 21:09:17.553993 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.553960    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:17.554161 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.554008    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:17.554161 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:17.554095    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:17.554503 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:17.554475    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:17.554590 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.554548    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:17.554672 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:17.554653    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:17.724372 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.724280    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nb5hx" event={"ID":"88e4b3aa-5e27-4609-ae83-e74875a7e8da","Type":"ContainerStarted","Data":"3040a31d4f7041202bd87375951f9bf1768df9b742eea11a63d68772b48b90b7"}
Oct 30 21:09:17.727055 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.727013    2568 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="17b34e55c4ea1b80af62ebf7a4ac76448f02aecca4b546c9e252b68c47aeb364" exitCode=0
Oct 30 21:09:17.728227 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.728203    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerDied","Data":"17b34e55c4ea1b80af62ebf7a4ac76448f02aecca4b546c9e252b68c47aeb364"}
Oct 30 21:09:17.766377 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.766322    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sm2w6" podStartSLOduration=3.642291025 podStartE2EDuration="11.766303424s" totalImagesPullingTime="8.124012399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:16.884019981 +0000 UTC m=+10.829765890" watchObservedRunningTime="2026-10-30 21:09:17.766303424 +0000 UTC m=+11.712049332"
Oct 30 21:09:17.766751 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:17.766715    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nb5hx" podStartSLOduration=3.659207336 podStartE2EDuration="11.766702436s" totalImagesPullingTime="8.1074951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:17.76646369 +0000 UTC m=+11.712209588" watchObservedRunningTime="2026-10-30 21:09:17.766702436 +0000 UTC m=+11.712448345"
Oct 30 21:09:18.730559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:18.730482    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-svkg2" event={"ID":"a2806653-74fd-4d37-a4ba-459232f41e74","Type":"ContainerStarted","Data":"454cec4c1cf902f7d5682fd2778f5483ca7ab6a03628ef0228b1072637acb866"}
Oct 30 21:09:18.733564 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:18.733531    2568 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="8c52aa936acc503ca6b37772ac6d2e868d6a496d2ce2a470d64fb1e1a2c15461" exitCode=0
Oct 30 21:09:18.734521 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:18.734266    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerDied","Data":"8c52aa936acc503ca6b37772ac6d2e868d6a496d2ce2a470d64fb1e1a2c15461"}
Oct 30 21:09:18.770466 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:18.770407    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-svkg2" podStartSLOduration=2.560530311 podStartE2EDuration="12.770392414s" totalImagesPullingTime="10.209862103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:18.769802368 +0000 UTC m=+12.715548275" watchObservedRunningTime="2026-10-30 21:09:18.770392414 +0000 UTC m=+12.716138320"
Oct 30 21:09:19.480840 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:19.480567    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bv86_93b394b4-d048-4139-a5eb-c32918e048d2/dns-node-resolver/0.log"
Oct 30 21:09:19.553866 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:19.553830    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:19.554053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:19.553830    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:19.554053 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:19.553951    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:19.554161 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:19.554057    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:19.554161 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:19.553847    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:19.554161 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:19.554133    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:20.632748 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:20.632725    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v7dh_67203193-932d-445f-b14d-457040dcee8e/node-ca/0.log"
Oct 30 21:09:20.749989 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:20.749944    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerStarted","Data":"9af294d17ef00892871c53310ae35b0abdb7a383931013d0545b964ef767ab2f"}
Oct 30 21:09:21.554114 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:21.554065    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:21.554114 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:21.554085    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:21.554336 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:21.554203    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:21.554336 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:21.554215    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:21.554336 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:21.554313    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:21.554482 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:21.554420    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:21.756103 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:21.756066    2568 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="9af294d17ef00892871c53310ae35b0abdb7a383931013d0545b964ef767ab2f" exitCode=0
Oct 30 21:09:21.756528 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:21.756112    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerDied","Data":"9af294d17ef00892871c53310ae35b0abdb7a383931013d0545b964ef767ab2f"}
Oct 30 21:09:22.761658 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.761385    2568 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="9946377acb35e3fea426c2217bb977b6e1534ecc1e2934b830b1b2a83f7cf1b6" exitCode=0
Oct 30 21:09:22.762154 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.761457    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerDied","Data":"9946377acb35e3fea426c2217bb977b6e1534ecc1e2934b830b1b2a83f7cf1b6"}
Oct 30 21:09:22.764022 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.763995    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:09:22.764368 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.764347    2568 generic.go:356] "Generic (PLEG): container finished" podID="054fd709-2d73-4ce0-ab94-0df9fbd1921a" containerID="b4ee4c31686215bbe112a5c83cd19b1745ea4beb85592d194434863d6e5ee34c" exitCode=1
Oct 30 21:09:22.764449 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.764375    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"f907554072593906ce1082ea24f41fb034c1be35f54656734ffc5d0f24661ced"}
Oct 30 21:09:22.764449 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.764440    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"944ef62a2c8074880b5d9470c16278331d5f9365e3a160bba2023e9d37dced1e"}
Oct 30 21:09:22.764507 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.764474    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerDied","Data":"b4ee4c31686215bbe112a5c83cd19b1745ea4beb85592d194434863d6e5ee34c"}
Oct 30 21:09:22.764507 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:22.764504    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"772d4fa17f92446f7ccefcf2a8382536dc159d12030bd87f52f97b6a7d863771"}
Oct 30 21:09:23.257187 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.257110    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:23.257187 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.257154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:23.257431 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.257256    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:23.257431 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.257258    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:23.257431 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.257305    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:39.257292201 +0000 UTC m=+33.203038086 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:23.257431 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.257318    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:39.257312529 +0000 UTC m=+33.203058413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:23.358329 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.358296    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:23.358529 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.358460    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:23.358529 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.358483    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:23.358529 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.358494    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:23.358639 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.358549    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:39.358534896 +0000 UTC m=+33.304280780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:23.474540 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.474504    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:23.474706 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.474668    2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:09:23.475163 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.475144    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:09:23.553960 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.553921    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:23.553960 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.553950    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:23.554212 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.554041    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:23.554212 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.554074    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:23.554212 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.554158    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:23.554347 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:23.554210    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:23.769569 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.769530    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k57t9" event={"ID":"19448ad0-e5a8-4760-b4c5-e8410e14af1b","Type":"ContainerStarted","Data":"e09e52bc591f042dbb63f978d2f223098a08bdd8255f74046451bef19e7995ac"}
Oct 30 21:09:23.772274 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.772252    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:09:23.772651 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.772630    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"292807454a64ce495c64ad7489ba3de3a3c933fc04f5f0496acf610f36a3ba9f"}
Oct 30 21:09:23.772721 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.772691    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"a691bec9f55d0a704701e1c713d8bf63076d3bdf4545f07740c181356fe6394a"}
Oct 30 21:09:23.814053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:23.813957    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k57t9" podStartSLOduration=7.500359177 podStartE2EDuration="17.813943842s" totalImagesPullingTime="10.313584665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:23.813886868 +0000 UTC m=+17.759632774" watchObservedRunningTime="2026-10-30 21:09:23.813943842 +0000 UTC m=+17.759689743"
Oct 30 21:09:25.553925 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:25.553873    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:25.554481 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:25.553893    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:25.554481 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:25.553990    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:25.554481 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:25.554005    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:25.554481 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:25.554107    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:25.554481 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:25.554204    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:25.779577 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:25.779550    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:09:25.779902 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:25.779876    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"03dac3f1b17177ccd8c3b98386097e6b0cc0e5ecddd7268d83b7ccf0ce414257"}
Oct 30 21:09:27.554006 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.553816    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:27.554529 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.553816    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:27.554529 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:27.554108    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:27.554529 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.553868    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:27.554529 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:27.554159    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:27.554529 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:27.554271    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:27.787054 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.787027    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:09:27.787412 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.787387    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"96050e0fb9205b76e77a1b640c275d0bf90e40fb98ca28373389d4a08f3cb5ae"}
Oct 30 21:09:27.787667 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.787654    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:27.787837 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.787819    2568 scope.go:122] "RemoveContainer" containerID="b4ee4c31686215bbe112a5c83cd19b1745ea4beb85592d194434863d6e5ee34c"
Oct 30 21:09:27.804875 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:27.804825    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:28.792798 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:28.792764    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:09:28.793329 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:28.793112    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" event={"ID":"054fd709-2d73-4ce0-ab94-0df9fbd1921a","Type":"ContainerStarted","Data":"227266c182a8c1d7edaeb5bd8d23713ba6179be709c898478005b457f337c036"}
Oct 30 21:09:28.793476 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:28.793458    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:28.793588 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:28.793557    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:28.808946 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:28.808922    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:09:28.877694 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:28.877643    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2" podStartSLOduration=8.297302331 podStartE2EDuration="22.877626405s" totalImagesPullingTime="14.580324074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:28.877392654 +0000 UTC m=+22.823138761" watchObservedRunningTime="2026-10-30 21:09:28.877626405 +0000 UTC m=+22.823372305"
Oct 30 21:09:29.554172 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:29.554136    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:29.554172 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:29.554169    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:29.554382 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:29.554179    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:29.554382 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:29.554267    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:29.554382 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:29.554366    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:29.554466 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:29.554432    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:31.553469 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:31.553434    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:31.553469 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:31.553462    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:31.554067 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:31.553534    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:31.554067 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:31.553597    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:31.554067 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:31.553727    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:31.554067 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:31.553821    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:33.553531 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:33.553476    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:33.553978 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:33.553479    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:33.553978 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:33.553632    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:33.553978 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:33.553594    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:33.553978 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:33.553700    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:33.553978 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:33.553765    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:35.554498 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:35.554453    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:35.555050 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:35.554589    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:35.555050 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:35.555017    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:35.555181 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:35.555110    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:35.555181 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:35.555164    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:35.555284 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:35.555249    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:36.269141 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:36.268823    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w46l"]
Oct 30 21:09:36.269306 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:36.269195    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:36.269349 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:36.269315    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:36.272551 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:36.272524    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pb8b2"]
Oct 30 21:09:36.272674 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:36.272656    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:36.272784 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:36.272763    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:36.280708 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:36.280677    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fbzn9"]
Oct 30 21:09:36.280863 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:36.280816    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:36.280957 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:36.280935    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:37.499782 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:37.499708    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bv86_93b394b4-d048-4139-a5eb-c32918e048d2/dns-node-resolver/0.log"
Oct 30 21:09:37.554259 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:37.554228    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:37.554414 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:37.554340    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:37.554414 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:37.554395    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:37.554508 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:37.554491    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:38.554132 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:38.554087    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:38.554553 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:38.554213    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w46l" podUID="3c61a330-f062-4cbf-9794-3f7973e3826d"
Oct 30 21:09:38.896472 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:38.896396    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v7dh_67203193-932d-445f-b14d-457040dcee8e/node-ca/0.log"
Oct 30 21:09:39.269390 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:39.269322    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:39.269390 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:39.269375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:39.269595 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.269472    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:39.269595 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.269473    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:39.269595 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.269525    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs podName:900d3f13-ab1c-478e-9851-a70264d92012 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:11.269510943 +0000 UTC m=+65.215256831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs") pod "network-metrics-daemon-fbzn9" (UID: "900d3f13-ab1c-478e-9851-a70264d92012") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:39.269595 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.269538    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret podName:3c61a330-f062-4cbf-9794-3f7973e3826d nodeName:}" failed. No retries permitted until 2026-10-30 21:10:11.269532545 +0000 UTC m=+65.215278430 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret") pod "global-pull-secret-syncer-8w46l" (UID: "3c61a330-f062-4cbf-9794-3f7973e3826d") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:39.370325 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:39.370284    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:39.370513 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.370434    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:39.370513 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.370455    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:39.370513 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.370467    2568 projected.go:196] Error preparing data for projected volume kube-api-access-q4r9r for pod openshift-network-diagnostics/network-check-target-pb8b2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:39.371155 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.370534    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r podName:38c2f0bf-d12e-497e-b64b-220761a147a6 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:11.370515185 +0000 UTC m=+65.316261084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q4r9r" (UniqueName: "kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r") pod "network-check-target-pb8b2" (UID: "38c2f0bf-d12e-497e-b64b-220761a147a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:39.553828 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:39.553749    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:39.553828 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:39.553792    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:39.554016 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.553864    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fbzn9" podUID="900d3f13-ab1c-478e-9851-a70264d92012"
Oct 30 21:09:39.554016 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:09:39.553970    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb8b2" podUID="38c2f0bf-d12e-497e-b64b-220761a147a6"
Oct 30 21:09:40.396777 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.396754    2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeReady"
Oct 30 21:09:40.397293 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.396899    2568 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:09:40.458815 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.458785    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zwsqg"]
Oct 30 21:09:40.510793 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.510759    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-slnfx"]
Oct 30 21:09:40.510953 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.510886    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.525540 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.525518    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:09:40.525681 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.525586    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:09:40.525681 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.525592    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6qvgx\""
Oct 30 21:09:40.531966 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.531943    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7tff6"]
Oct 30 21:09:40.532056 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.531997    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.548249 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.548232    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zwsqg"]
Oct 30 21:09:40.548346 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.548262    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slnfx"]
Oct 30 21:09:40.548392 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.548357    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.548392 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.548385    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7tff6"]
Oct 30 21:09:40.553409 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.553394    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:09:40.577135 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.577115    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca7df3d-5d59-436f-a381-a0224e48272f-config-volume\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.577224 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.577159    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6n7c\" (UniqueName: \"kubernetes.io/projected/7baeebaf-f8c0-4643-b921-fff75980046a-kube-api-access-f6n7c\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.577282 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.577267    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca7df3d-5d59-436f-a381-a0224e48272f-tmp-dir\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.577314 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.577296    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fl2\" (UniqueName: \"kubernetes.io/projected/2ca7df3d-5d59-436f-a381-a0224e48272f-kube-api-access-s9fl2\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.577350 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.577328    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ca7df3d-5d59-436f-a381-a0224e48272f-metrics-tls\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.577396 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.577384    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7baeebaf-f8c0-4643-b921-fff75980046a-cert\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.625898 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.625876    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:09:40.625995 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.625962    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w6j9l\""
Oct 30 21:09:40.625995 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.625971    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:09:40.625995 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.625968    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:09:40.626104 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.625995    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:09:40.626104 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.626027    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:09:40.626104 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.626064    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:09:40.626199 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.626115    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8rdf8\""
Oct 30 21:09:40.626199 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.626170    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\""
Oct 30 21:09:40.626264 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.626224    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:09:40.626264 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.626262    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:09:40.678655 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678624    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fl2\" (UniqueName: \"kubernetes.io/projected/2ca7df3d-5d59-436f-a381-a0224e48272f-kube-api-access-s9fl2\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.678770 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678667    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-data-volume\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.678770 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trphj\" (UniqueName: \"kubernetes.io/projected/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-api-access-trphj\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.678839 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ca7df3d-5d59-436f-a381-a0224e48272f-metrics-tls\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.678839 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678791    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-crio-socket\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.678839 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678811    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.678955 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.678871    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7baeebaf-f8c0-4643-b921-fff75980046a-cert\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.679077 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.679060    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca7df3d-5d59-436f-a381-a0224e48272f-config-volume\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.679136 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.679121    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6n7c\" (UniqueName: \"kubernetes.io/projected/7baeebaf-f8c0-4643-b921-fff75980046a-kube-api-access-f6n7c\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.679189 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.679176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca7df3d-5d59-436f-a381-a0224e48272f-tmp-dir\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.679225 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.679206    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.679622 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.679581    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca7df3d-5d59-436f-a381-a0224e48272f-tmp-dir\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.679622 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.679596    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca7df3d-5d59-436f-a381-a0224e48272f-config-volume\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.682614 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.682582    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ca7df3d-5d59-436f-a381-a0224e48272f-metrics-tls\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.682701 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.682645    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7baeebaf-f8c0-4643-b921-fff75980046a-cert\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.699963 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.699939    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6n7c\" (UniqueName: \"kubernetes.io/projected/7baeebaf-f8c0-4643-b921-fff75980046a-kube-api-access-f6n7c\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.700268 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.700250    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fl2\" (UniqueName: \"kubernetes.io/projected/2ca7df3d-5d59-436f-a381-a0224e48272f-kube-api-access-s9fl2\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.780419 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.780551 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780475    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.780551 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780512    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-data-volume\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.780551 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780533    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trphj\" (UniqueName: \"kubernetes.io/projected/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-api-access-trphj\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.780687 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780559    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-crio-socket\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.780774 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780757    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-crio-socket\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.780904 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.780878    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-data-volume\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.781131 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.781114    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.782589 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.782570    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.800488 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.800468    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trphj\" (UniqueName: \"kubernetes.io/projected/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-api-access-trphj\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:40.855456 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.855386    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:40.872708 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.872683    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:09:40.926963 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:40.926936    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:09:41.249888 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.249789    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zwsqg"]
Oct 30 21:09:41.252993 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.252949    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slnfx"]
Oct 30 21:09:41.255976 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:41.255947    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca7df3d_5d59_436f_a381_a0224e48272f.slice/crio-89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f WatchSource:0}: Error finding container 89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f: Status 404 returned error can't find the container with id 89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f
Oct 30 21:09:41.257660 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:41.257640    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7baeebaf_f8c0_4643_b921_fff75980046a.slice/crio-7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614 WatchSource:0}: Error finding container 7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614: Status 404 returned error can't find the container with id 7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614
Oct 30 21:09:41.265035 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.265011    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7tff6"]
Oct 30 21:09:41.270706 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:41.270668    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c05ea68_c44a_4fbc_bda8_e6cb9ea15327.slice/crio-a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58 WatchSource:0}: Error finding container a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58: Status 404 returned error can't find the container with id a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58
Oct 30 21:09:41.554143 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.554110    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:09:41.554687 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.554121    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:09:41.568860 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.568838    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:09:41.568982 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.568948    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:09:41.569049 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.569011    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\""
Oct 30 21:09:41.569101 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.569083    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:09:41.569153 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.569128    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\""
Oct 30 21:09:41.824818 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.824714    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slnfx" event={"ID":"7baeebaf-f8c0-4643-b921-fff75980046a","Type":"ContainerStarted","Data":"7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614"}
Oct 30 21:09:41.827667 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.827631    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwsqg" event={"ID":"2ca7df3d-5d59-436f-a381-a0224e48272f","Type":"ContainerStarted","Data":"89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f"}
Oct 30 21:09:41.831401 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.831357    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tff6" event={"ID":"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327","Type":"ContainerStarted","Data":"96defc20640a7af6c35aff13ceb40c8a902ff7cb209d1df9f547128b1c5ee528"}
Oct 30 21:09:41.831559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:41.831424    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tff6" event={"ID":"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327","Type":"ContainerStarted","Data":"a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58"}
Oct 30 21:09:42.457896 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.457867    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"]
Oct 30 21:09:42.483370 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.483131    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"]
Oct 30 21:09:42.483370 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.483273    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.500207 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.497085    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"]
Oct 30 21:09:42.506837 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.506806    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:09:42.506977 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.506853    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:09:42.512487 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.512421    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 21:09:42.512578 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.512483    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:09:42.512659 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.512591    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2rp7h\""
Oct 30 21:09:42.514091 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.514071    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.521790 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.521758    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8tkj4"]
Oct 30 21:09:42.533159 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.533029    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:09:42.533159 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.533146    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wc7hh\""
Oct 30 21:09:42.533773 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.533730    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:09:42.542793 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.542761    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"]
Oct 30 21:09:42.542910 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.542896    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.562777 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.562737    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:09:42.563180 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.562863    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28pwt\""
Oct 30 21:09:42.563180 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.562874    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:09:42.575845 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.575797    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:09:42.575845 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.575840    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:09:42.576049 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.575976    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:09:42.592866 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.592834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7twc\" (UniqueName: \"kubernetes.io/projected/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-kube-api-access-j7twc\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593027 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.592883    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-sys\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593027 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.592978    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9br6h\" (UniqueName: \"kubernetes.io/projected/b3d13165-1255-4e98-b0c8-a84b78043c93-kube-api-access-9br6h\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.593131 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593040    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593131 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593073    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-metrics-client-ca\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593131 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593099    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjxk\" (UniqueName: \"kubernetes.io/projected/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-api-access-tkjxk\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.593233 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593128    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.593233 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593155    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.593233 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593178    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d13165-1255-4e98-b0c8-a84b78043c93-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.593233 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593202    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.593233 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593227    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.593406 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-textfile\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593406 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593265    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.593406 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593287    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.593406 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-root\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593406 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593367    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-tls\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593561 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593473    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.593561 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593530    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-wtmp\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.593672 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.593559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.694966 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.694881    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-wtmp\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.694966 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.694931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695196 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.694978    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7twc\" (UniqueName: \"kubernetes.io/projected/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-kube-api-access-j7twc\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695196 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695008    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-sys\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695196 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9br6h\" (UniqueName: \"kubernetes.io/projected/b3d13165-1255-4e98-b0c8-a84b78043c93-kube-api-access-9br6h\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.695196 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695065    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-wtmp\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695196 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695077    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-sys\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695214    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-metrics-client-ca\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695282    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjxk\" (UniqueName: \"kubernetes.io/projected/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-api-access-tkjxk\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.695400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695312    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.695400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695341    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.695400 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d13165-1255-4e98-b0c8-a84b78043c93-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.695585 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695455    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.695585 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695500    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.695585 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-textfile\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695585 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695546    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695585 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695556    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.695789 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695587    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.695789 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695689    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-root\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695789 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695715    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-tls\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.695789 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.695750    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.696085 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.696059    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-metrics-client-ca\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.696268 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.696243    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d13165-1255-4e98-b0c8-a84b78043c93-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.696513 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.696489    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.696623 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.696581    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.696894 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.696855    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-textfile\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.696894 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.696860    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-root\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.697461 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.697412    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.700053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.699761    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.700053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.699907    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.700053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.700012    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.700053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.700024    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.701169 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.701150    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.701497 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.701475    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-tls\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.724041 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.724009    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7twc\" (UniqueName: \"kubernetes.io/projected/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-kube-api-access-j7twc\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.724366 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.724328    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9br6h\" (UniqueName: \"kubernetes.io/projected/b3d13165-1255-4e98-b0c8-a84b78043c93-kube-api-access-9br6h\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.726479 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.726451    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjxk\" (UniqueName: \"kubernetes.io/projected/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-api-access-tkjxk\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.836949 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.836904    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tff6" event={"ID":"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327","Type":"ContainerStarted","Data":"21d6985b21738a915b1c3cb0489eff4ac8b38eafa23e25b60c44ef4da1491a28"}
Oct 30 21:09:42.853369 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.853333    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:09:42.925861 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.925591    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:09:42.937812 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:42.937776    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:09:42.976616 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:42.976569    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8846e2f3_9d46_4bf1_a71e_2f9f09e6d85e.slice/crio-1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a WatchSource:0}: Error finding container 1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a: Status 404 returned error can't find the container with id 1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a
Oct 30 21:09:43.215475 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.215401    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"]
Oct 30 21:09:43.255215 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.255173    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"]
Oct 30 21:09:43.263562 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:43.263512    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d13165_1255_4e98_b0c8_a84b78043c93.slice/crio-438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca WatchSource:0}: Error finding container 438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca: Status 404 returned error can't find the container with id 438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca
Oct 30 21:09:43.271751 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:43.271719    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9f385c_ce7e_42d9_a79c_2ef0976f44f5.slice/crio-5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547 WatchSource:0}: Error finding container 5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547: Status 404 returned error can't find the container with id 5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547
Oct 30 21:09:43.842061 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.842020    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwsqg" event={"ID":"2ca7df3d-5d59-436f-a381-a0224e48272f","Type":"ContainerStarted","Data":"3ea4e6d38f73a761e2ba7f24dcf80d1ae7a42531932698b71f0b8f92a8ff3561"}
Oct 30 21:09:43.845089 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.845047    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2" event={"ID":"b3d13165-1255-4e98-b0c8-a84b78043c93","Type":"ContainerStarted","Data":"b9e065fc1088aaf241b95e85b26e4a1a46f4a4959584c6a58221b9022cede46d"}
Oct 30 21:09:43.845219 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.845118    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2" event={"ID":"b3d13165-1255-4e98-b0c8-a84b78043c93","Type":"ContainerStarted","Data":"5233e554956d2396cfa66b22130bd14e964ffeb10e4cfd81987e55ce69df6fe0"}
Oct 30 21:09:43.845219 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.845146    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2" event={"ID":"b3d13165-1255-4e98-b0c8-a84b78043c93","Type":"ContainerStarted","Data":"438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca"}
Oct 30 21:09:43.846813 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.846782    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8tkj4" event={"ID":"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e","Type":"ContainerStarted","Data":"1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a"}
Oct 30 21:09:43.848262 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.848237    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg" event={"ID":"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5","Type":"ContainerStarted","Data":"5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547"}
Oct 30 21:09:43.935688 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.935494    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:09:43.952070 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.952042    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:43.964046 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.963595    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:09:43.967150 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.966960    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:09:43.967150 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.967041    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:09:43.974595 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974556    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:09:43.974759 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974650    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:09:43.974759 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974675    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:09:43.974759 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974710    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:09:43.974759 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974566    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:09:43.974759 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974758    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:09:43.974977 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.974769    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-knzzg\""
Oct 30 21:09:43.990065 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:43.990023    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009283    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009466    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009491    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009517    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009546    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009574    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009640    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009677    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009713    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.010125 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.009792    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111139 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111108    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111139 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111144    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111223    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111246    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111269    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111294    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111322    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111359 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111373    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111398    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111431    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.111559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.111458    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.113008 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.112330    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.113008 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.112432    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.113008 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.112508    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.117634 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.117582    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.117753 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.117674    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.117753 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.117703    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.118140 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.118116    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.118351 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.118330    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.118444 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.118346    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.118810 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.118786    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.118985 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.118929    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.120284 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.120261    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.130671 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.130638    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.337526 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.337494    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:09:44.747405 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.747316    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"]
Oct 30 21:09:44.771445 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.771413    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:09:44.771568 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.771532    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.777241 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.776863    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"]
Oct 30 21:09:44.785808 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.785779    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:09:44.785969 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.785953    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s8jwz\""
Oct 30 21:09:44.786087 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.786070    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:09:44.786152 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.786138    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:09:44.786152 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.786143    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:09:44.786325 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.786311    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:09:44.786384 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.786350    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bcg3387hou02k\""
Oct 30 21:09:44.863735 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.863692    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwsqg" event={"ID":"2ca7df3d-5d59-436f-a381-a0224e48272f","Type":"ContainerStarted","Data":"6182dfb7cbcd5078334c475156db45095dced3f145245aa99c560413632f615b"}
Oct 30 21:09:44.866186 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:44.866154    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06a95f5_9691_4b5d_b247_1c2c0d524f5d.slice/crio-f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991 WatchSource:0}: Error finding container f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991: Status 404 returned error can't find the container with id f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991
Oct 30 21:09:44.878785 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.878755    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:44.919234 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.919187    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zwsqg" podStartSLOduration=2.491162485 podStartE2EDuration="4.919173013s" totalImagesPullingTime="2.428010528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:44.918146596 +0000 UTC m=+38.863892503" watchObservedRunningTime="2026-10-30 21:09:44.919173013 +0000 UTC m=+38.864918916"
Oct 30 21:09:44.919714 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.919688    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb96b\" (UniqueName: \"kubernetes.io/projected/67790f00-7ce5-4f74-ae11-893cf8e217ad-kube-api-access-pb96b\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.919829 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.919733    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.919829 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.919818    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67790f00-7ce5-4f74-ae11-893cf8e217ad-tmp\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.919917 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.919874    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67790f00-7ce5-4f74-ae11-893cf8e217ad-metrics-client-ca\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.920007 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.919984    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.920085 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.920068    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.920195 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.920173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.920277 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.920211    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:44.920334 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:44.920280    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-grpc-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.020855 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.020818    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021027 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.020869    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021027 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.020892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-grpc-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021027 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.020992    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb96b\" (UniqueName: \"kubernetes.io/projected/67790f00-7ce5-4f74-ae11-893cf8e217ad-kube-api-access-pb96b\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021146 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.021026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021146 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.021050    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67790f00-7ce5-4f74-ae11-893cf8e217ad-tmp\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021445 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.021417    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67790f00-7ce5-4f74-ae11-893cf8e217ad-tmp\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021736 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.021680    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67790f00-7ce5-4f74-ae11-893cf8e217ad-metrics-client-ca\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021831 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.021740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.021831 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.021790    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.022379 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.022352    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67790f00-7ce5-4f74-ae11-893cf8e217ad-metrics-client-ca\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.025140 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.025112    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-grpc-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.025706 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.025579    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.025706 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.025629    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.025706 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.025677    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.025853 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.025743    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.025853 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.025756    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.041190 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.041159    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb96b\" (UniqueName: \"kubernetes.io/projected/67790f00-7ce5-4f74-ae11-893cf8e217ad-kube-api-access-pb96b\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.179920 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.179758    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:45.547520 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.547312    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"]
Oct 30 21:09:45.554085 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:45.554055    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67790f00_7ce5_4f74_ae11_893cf8e217ad.slice/crio-23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2 WatchSource:0}: Error finding container 23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2: Status 404 returned error can't find the container with id 23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2
Oct 30 21:09:45.868442 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.868411    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2"}
Oct 30 21:09:45.873252 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.873157    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tff6" event={"ID":"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327","Type":"ContainerStarted","Data":"78605b3f351d8a3a17c473c1afd9634e80543b70e2a7f49928a2c47c82dccba2"}
Oct 30 21:09:45.876693 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.876661    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2" event={"ID":"b3d13165-1255-4e98-b0c8-a84b78043c93","Type":"ContainerStarted","Data":"12051c03579a41de4cd4590697eea1a5fc124f8d2defba284d721c33eacc2130"}
Oct 30 21:09:45.878134 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.878110    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991"}
Oct 30 21:09:45.879676 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.879654    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slnfx" event={"ID":"7baeebaf-f8c0-4643-b921-fff75980046a","Type":"ContainerStarted","Data":"9f22fe915d011cf0cb9b4fd9a89eeb5fe0c2ab31ae9370dc6e0f6826b199785c"}
Oct 30 21:09:45.882568 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.881948    2568 generic.go:356] "Generic (PLEG): container finished" podID="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" containerID="5d9fdca04fc0bbbc4750fd5cc543405a0f7d336ac4c65a42d3f8f5937fe68f77" exitCode=0
Oct 30 21:09:45.882568 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.882029    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8tkj4" event={"ID":"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e","Type":"ContainerDied","Data":"5d9fdca04fc0bbbc4750fd5cc543405a0f7d336ac4c65a42d3f8f5937fe68f77"}
Oct 30 21:09:45.885215 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.885189    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg" event={"ID":"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5","Type":"ContainerStarted","Data":"464a81337420b3e26635b6023c843ca4d73f571601b40711d06035af41b250d8"}
Oct 30 21:09:45.920128 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.914700    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7tff6" podStartSLOduration=2.230599932 podStartE2EDuration="5.914676359s" totalImagesPullingTime="3.684076427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:45.912911404 +0000 UTC m=+39.858657311" watchObservedRunningTime="2026-10-30 21:09:45.914676359 +0000 UTC m=+39.860422266"
Oct 30 21:09:45.955406 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:45.955346    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-slnfx" podStartSLOduration=2.112372012 podStartE2EDuration="5.955325316s" totalImagesPullingTime="3.842953304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:45.953550124 +0000 UTC m=+39.899296028" watchObservedRunningTime="2026-10-30 21:09:45.955325316 +0000 UTC m=+39.901071227"
Oct 30 21:09:46.048113 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.048053    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2" podStartSLOduration=2.261447628 podStartE2EDuration="4.048028843s" totalImagesPullingTime="1.786581215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.048016243 +0000 UTC m=+39.993762153" watchObservedRunningTime="2026-10-30 21:09:46.048028843 +0000 UTC m=+39.993774764"
Oct 30 21:09:46.892360 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.892324    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8tkj4" event={"ID":"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e","Type":"ContainerStarted","Data":"15ab2ad39cfe7c8c80d277ddef2f5889dd2f53d8edcf5f5c1a8c45ce2be191fa"}
Oct 30 21:09:46.892837 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.892392    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8tkj4" event={"ID":"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e","Type":"ContainerStarted","Data":"15fa88e3d3d9d9a39ada1411fdae351aa838b33f60a75d48d314046b3a0861a1"}
Oct 30 21:09:46.895413 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.895380    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg" event={"ID":"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5","Type":"ContainerStarted","Data":"18ba0385e8fd4c8406b3ecea7dc92eeb2fd44cf370f451ac16ed588c19ffb06e"}
Oct 30 21:09:46.895555 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.895457    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg" event={"ID":"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5","Type":"ContainerStarted","Data":"082d472bad1245ac984077800bfcc4d71f0a8dfccc5474ffcc498352d63bae62"}
Oct 30 21:09:46.897205 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.897178    2568 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1" exitCode=0
Oct 30 21:09:46.897545 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.897524    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1"}
Oct 30 21:09:46.931580 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:46.931487    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8tkj4" podStartSLOduration=2.638905393 podStartE2EDuration="4.931473072s" totalImagesPullingTime="2.183232227s" totalInitContainerRuntime="109.335452ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.930836557 +0000 UTC m=+40.876582463" watchObservedRunningTime="2026-10-30 21:09:46.931473072 +0000 UTC m=+40.877218975"
Oct 30 21:09:47.005404 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.005351    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg" podStartSLOduration=2.585629393 podStartE2EDuration="5.005337138s" totalImagesPullingTime="2.419707745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:46.969083816 +0000 UTC m=+40.914829746" watchObservedRunningTime="2026-10-30 21:09:47.005337138 +0000 UTC m=+40.951083045"
Oct 30 21:09:47.005678 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.005663    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54db77678f-lbdmq"]
Oct 30 21:09:47.033880 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.033842    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54db77678f-lbdmq"]
Oct 30 21:09:47.034069 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.033987    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.049231 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.049199    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:09:47.049231 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.049199    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:09:47.049401 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.049247    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kb9ct\""
Oct 30 21:09:47.049401 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.049281    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:09:47.049401 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.049307    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8dcb97lchjofv\""
Oct 30 21:09:47.049401 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.049329    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:09:47.142460 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142420    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.142661 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-metrics-server-audit-profiles\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.142661 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2eed9c68-fae8-40b3-9bee-499e396d96e5-audit-log\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.142771 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142672    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s697g\" (UniqueName: \"kubernetes.io/projected/2eed9c68-fae8-40b3-9bee-499e396d96e5-kube-api-access-s697g\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.142771 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142729    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-client-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.142852 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142805    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-tls\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.142898 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.142852    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-client-certs\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.243956 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.243870    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s697g\" (UniqueName: \"kubernetes.io/projected/2eed9c68-fae8-40b3-9bee-499e396d96e5-kube-api-access-s697g\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.243956 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.243927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-client-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.244177 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.243970    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-tls\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.244177 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.244011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-client-certs\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.244177 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.244056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.244177 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.244117    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-metrics-server-audit-profiles\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.244177 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.244176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2eed9c68-fae8-40b3-9bee-499e396d96e5-audit-log\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.244554 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.244525    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2eed9c68-fae8-40b3-9bee-499e396d96e5-audit-log\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.245003 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.244974    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.245335 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.245307    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-metrics-server-audit-profiles\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.247557 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.247534    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-client-certs\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.247724 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.247707    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-tls\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.247820 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.247760    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-client-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.265664 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.265630    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s697g\" (UniqueName: \"kubernetes.io/projected/2eed9c68-fae8-40b3-9bee-499e396d96e5-kube-api-access-s697g\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.325128 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.325089    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"]
Oct 30 21:09:47.355827 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.355787    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"]
Oct 30 21:09:47.355827 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.355814    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:09:47.356075 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.355820    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.371053 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.371020    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:09:47.371244 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.371162    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-89vxb\""
Oct 30 21:09:47.446297 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.446264    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34a0ac7b-1822-4157-9438-d76e6783177e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.446454 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.446383    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgkt\" (UniqueName: \"kubernetes.io/projected/34a0ac7b-1822-4157-9438-d76e6783177e-kube-api-access-5hgkt\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.547790 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.547742    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34a0ac7b-1822-4157-9438-d76e6783177e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.547987 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.547854    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgkt\" (UniqueName: \"kubernetes.io/projected/34a0ac7b-1822-4157-9438-d76e6783177e-kube-api-access-5hgkt\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.551767 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.551740    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34a0ac7b-1822-4157-9438-d76e6783177e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.567887 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.567853    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgkt\" (UniqueName: \"kubernetes.io/projected/34a0ac7b-1822-4157-9438-d76e6783177e-kube-api-access-5hgkt\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.680060 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.680022    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:47.689142 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.689108    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54db77678f-lbdmq"]
Oct 30 21:09:47.695974 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:47.695944    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eed9c68_fae8_40b3_9bee_499e396d96e5.slice/crio-6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02 WatchSource:0}: Error finding container 6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02: Status 404 returned error can't find the container with id 6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02
Oct 30 21:09:47.904946 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:47.904644    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq" event={"ID":"2eed9c68-fae8-40b3-9bee-499e396d96e5","Type":"ContainerStarted","Data":"6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02"}
Oct 30 21:09:48.002764 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.002729    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"]
Oct 30 21:09:48.102988 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:48.102948    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a0ac7b_1822_4157_9438_d76e6783177e.slice/crio-7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814 WatchSource:0}: Error finding container 7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814: Status 404 returned error can't find the container with id 7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814
Oct 30 21:09:48.106747 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.106716    2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85d787b45d-5wm46"]
Oct 30 21:09:48.109029 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.109009    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.123832 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.123812    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:09:48.126898 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.126873    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:09:48.127049 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.127027    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:09:48.127260 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.127230    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:09:48.127357 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.127295    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:09:48.127357 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.127341    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:09:48.129323 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.129301    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d787b45d-5wm46"]
Oct 30 21:09:48.135825 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.135802    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:09:48.151598 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.151553    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:09:48.226498 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.226439    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v2rbx\""
Oct 30 21:09:48.256450 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256414    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256581 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256466    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256581 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256503    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256581 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256531    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256581 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256563    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256809 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256593    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256809 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256650    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.256809 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.256677    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357415 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357590 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357438    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357590 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357481    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357590 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357590 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357570    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357850 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357599    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357850 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357850 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357718    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.357991 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.357873    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.358287 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.358259    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.358658 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.358566    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.358658 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.358575    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.359249 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.359224    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.361532 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.361508    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.361861 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.361838    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.383813 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.383786    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.422701 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.422660    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:09:48.728516 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.728479    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d787b45d-5wm46"]
Oct 30 21:09:48.737118 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:09:48.737079    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a6179c_e8ae_4870_a41e_29e65a6373e1.slice/crio-8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007 WatchSource:0}: Error finding container 8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007: Status 404 returned error can't find the container with id 8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007
Oct 30 21:09:48.913868 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.913659    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"ad0acf2afc32e8a264ac3f74e224c783ee131b37c1150e7de9a8ed4ae038530d"}
Oct 30 21:09:48.913868 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.913745    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"481465390c203e0e6ef50af4ad75451550212bc2ef60516e61f26c0278121f7a"}
Oct 30 21:09:48.913868 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.913783    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"3c6b4b36a56d54a7dda2412be066d5d1f197d1f8714b6951fd45d892cd9dc9f0"}
Oct 30 21:09:48.917318 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.917289    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh" event={"ID":"34a0ac7b-1822-4157-9438-d76e6783177e","Type":"ContainerStarted","Data":"7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814"}
Oct 30 21:09:48.921781 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.921749    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2"}
Oct 30 21:09:48.921929 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.921821    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9"}
Oct 30 21:09:48.921929 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.921849    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261"}
Oct 30 21:09:48.921929 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.921875    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075"}
Oct 30 21:09:48.922965 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:48.922924    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d787b45d-5wm46" event={"ID":"71a6179c-e8ae-4870-a41e-29e65a6373e1","Type":"ContainerStarted","Data":"8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007"}
Oct 30 21:09:49.932203 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.932017    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768"}
Oct 30 21:09:49.932640 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.932237    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerStarted","Data":"4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab"}
Oct 30 21:09:49.937559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.936510    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"75a7e7dce647c818ffdf27289854037f7c26c0ffdedfb9b5451bc70fabb9a91c"}
Oct 30 21:09:49.937559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.936590    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"846cc8b73bed64ffb829c9d1c44f24ffe0135e37bf0e27b7cf92c9d2b12576ad"}
Oct 30 21:09:49.937559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.936645    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" event={"ID":"67790f00-7ce5-4f74-ae11-893cf8e217ad","Type":"ContainerStarted","Data":"6033dd9699d8d728f5b5918f717243d6bca59c06e2067a46926dcf7fe0ec3d8a"}
Oct 30 21:09:49.937559 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.937474    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:49.939711 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.939676    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh" event={"ID":"34a0ac7b-1822-4157-9438-d76e6783177e","Type":"ContainerStarted","Data":"60222cf2d5edea74b45d8a225f0b19fbf13f7b8c60952354747cf2387bea1f5b"}
Oct 30 21:09:49.940494 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.940472    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:49.947301 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.947278    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:09:49.992713 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:49.992646    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.911071293 podStartE2EDuration="6.992627168s" totalImagesPullingTime="4.055068519s" totalInitContainerRuntime="26.487356ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:49.991410246 +0000 UTC m=+43.937156195" watchObservedRunningTime="2026-10-30 21:09:49.992627168 +0000 UTC m=+43.938373078"
Oct 30 21:09:50.036042 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:50.035975    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb" podStartSLOduration=2.38467121 podStartE2EDuration="6.035954809s" totalImagesPullingTime="3.651283599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:50.03500081 +0000 UTC m=+43.980746714" watchObservedRunningTime="2026-10-30 21:09:50.035954809 +0000 UTC m=+43.981700717"
Oct 30 21:09:50.076266 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:50.076128    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh" podStartSLOduration=1.386257512 podStartE2EDuration="3.076108222s" totalImagesPullingTime="1.68985071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:50.073042215 +0000 UTC m=+44.018788123" watchObservedRunningTime="2026-10-30 21:09:50.076108222 +0000 UTC m=+44.021854131"
Oct 30 21:09:50.946488 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:50.946445    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq" event={"ID":"2eed9c68-fae8-40b3-9bee-499e396d96e5","Type":"ContainerStarted","Data":"94be5581397f6f4a1d4ef18923922626278ca3f5cac954e0d63680083242cbca"}
Oct 30 21:09:50.992946 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:50.992884    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq" podStartSLOduration=2.508995508 podStartE2EDuration="4.992870584s" totalImagesPullingTime="2.483875076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:50.991981881 +0000 UTC m=+44.937727789" watchObservedRunningTime="2026-10-30 21:09:50.992870584 +0000 UTC m=+44.938616491"
Oct 30 21:09:51.952168 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:51.952133    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d787b45d-5wm46" event={"ID":"71a6179c-e8ae-4870-a41e-29e65a6373e1","Type":"ContainerStarted","Data":"6548df44376feac9377eadb80dee7c214556a7ed6ed983c5c6c3820ff153360d"}
Oct 30 21:09:54.887764 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:54.887729    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:09:54.926281 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:54.926227    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85d787b45d-5wm46" podStartSLOduration=3.934238279 podStartE2EDuration="6.926211633s" totalImagesPullingTime="2.991973354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:52.005648808 +0000 UTC m=+45.951394711" watchObservedRunningTime="2026-10-30 21:09:54.926211633 +0000 UTC m=+48.871957544"
Oct 30 21:09:56.959797 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:56.959764    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:09:57.947151 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:57.947115    2568 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d787b45d-5wm46"]
Oct 30 21:09:58.423329 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:09:58.423286    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:00.808240 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:00.808213    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:07.356856 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:07.356805    2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:07.357420 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:07.356883    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:10.954148 ip-10-0-142-232 kubenswrapper[2568]: E1030 21:10:10.954111    2568 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:10:11.269771 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.269742    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:11.269947 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.269811    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:11.285383 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.285350    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:10:11.293346 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.293312    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:11.297336 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.297315    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:10:11.302495 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.302474    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:11.312988 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.312964    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\""
Oct 30 21:10:11.320080 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.320060    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:11.371234 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.371205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:11.386131 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.386096    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:10:11.405279 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.405246    2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:10:11.414705 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.414680    2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:11.527049 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.526970    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\""
Oct 30 21:10:11.533097 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.533076    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:11.591364 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.591336    2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\""
Oct 30 21:10:11.598490 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.598222    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:11.619929 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.619895    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fbzn9"]
Oct 30 21:10:11.624337 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:10:11.624307    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900d3f13_ab1c_478e_9851_a70264d92012.slice/crio-85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba WatchSource:0}: Error finding container 85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba: Status 404 returned error can't find the container with id 85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba
Oct 30 21:10:11.830785 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.830701    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w46l"]
Oct 30 21:10:11.838106 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:10:11.838073    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c61a330_f062_4cbf_9794_3f7973e3826d.slice/crio-ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561 WatchSource:0}: Error finding container ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561: Status 404 returned error can't find the container with id ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561
Oct 30 21:10:11.906420 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:11.906381    2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pb8b2"]
Oct 30 21:10:11.911552 ip-10-0-142-232 kubenswrapper[2568]: W1030 21:10:11.911519    2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c2f0bf_d12e_497e_b64b_220761a147a6.slice/crio-14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83 WatchSource:0}: Error finding container 14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83: Status 404 returned error can't find the container with id 14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83
Oct 30 21:10:12.027295 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:12.027257    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fbzn9" event={"ID":"900d3f13-ab1c-478e-9851-a70264d92012","Type":"ContainerStarted","Data":"85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba"}
Oct 30 21:10:12.028441 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:12.028412    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w46l" event={"ID":"3c61a330-f062-4cbf-9794-3f7973e3826d","Type":"ContainerStarted","Data":"ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561"}
Oct 30 21:10:12.029649 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:12.029619    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pb8b2" event={"ID":"38c2f0bf-d12e-497e-b64b-220761a147a6","Type":"ContainerStarted","Data":"14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83"}
Oct 30 21:10:13.037182 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:13.037143    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fbzn9" event={"ID":"900d3f13-ab1c-478e-9851-a70264d92012","Type":"ContainerStarted","Data":"d8a29c88430e16f2ce74b12ca49ee4fa7206e17f29d1bf3eb1865d56cf2a4250"}
Oct 30 21:10:13.037717 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:13.037198    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fbzn9" event={"ID":"900d3f13-ab1c-478e-9851-a70264d92012","Type":"ContainerStarted","Data":"402ef58dd3c5d3403971c2f18d393a5ed3a290a36c387d6d8a864b4f6413f708"}
Oct 30 21:10:16.052189 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:16.052142    2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pb8b2" event={"ID":"38c2f0bf-d12e-497e-b64b-220761a147a6","Type":"ContainerStarted","Data":"7d6a6a7494224c403fe3c727a4e3ceb443d392ae1f5d395a1d1ca7afeaf6f516"}
Oct 30 21:10:16.052690 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:16.052296    2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:16.089486 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:16.089421    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fbzn9" podStartSLOduration=69.084495176 podStartE2EDuration="1m10.089401989s" totalImagesPullingTime="1.004906813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:13.075585815 +0000 UTC m=+67.021331730" watchObservedRunningTime="2026-10-30 21:10:16.089401989 +0000 UTC m=+70.035147902"
Oct 30 21:10:16.090558 ip-10-0-142-232 kubenswrapper[2568]: I1030 21:10:16.090514    2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pb8b2" podStartSLOduration=66.720693628 podStartE2EDuration="1m10.090501508s" totalImagesPullingTime="3.36980788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:16.088440833 +0000 UTC m=+70.034186741" watchObservedRunningTime="2026-10-30 21:10:16.090501508 +0000 UTC m=+70.036247414"
Oct 30 21:10:17.022103 ip-10-0-142-232 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:10:17.030423 ip-10-0-142-232 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:10:17.030593 ip-10-0-142-232 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:10:17.030790 ip-10-0-142-232 systemd[1]: kubelet.service: Consumed 4.571s CPU time.
Oct 30 21:10:17.045244 ip-10-0-142-232 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:10:17.119786 ip-10-0-142-232 kubenswrapper[6695]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:10:17.119786 ip-10-0-142-232 kubenswrapper[6695]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:10:17.119786 ip-10-0-142-232 kubenswrapper[6695]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:10:17.119786 ip-10-0-142-232 kubenswrapper[6695]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:10:17.121816 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121718    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.121816 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121815    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.121816 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121819    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.121816 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121822    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121825    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121828    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121831    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121834    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121836    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121843    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121846    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121849    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121851    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121854    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121856    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121859    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121862    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121864    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121867    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121869    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121872    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121875    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121878    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.121953 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121881    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121883    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121886    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121889    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121891    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121896    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121901    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121905    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121908    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121911    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121914    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121917    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121919    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121922    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121925    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121927    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121931    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121933    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121936    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.122460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121938    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121941    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121945    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121949    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121952    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121955    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121958    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121961    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121964    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121967    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121969    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121972    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121974    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121977    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121980    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121983    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121985    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121988    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121991    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.122949 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121994    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.121997    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122001    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122003    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122006    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122008    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122011    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122014    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122016    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122019    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122021    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122024    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122027    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122029    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122032    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122034    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122037    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122039    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122042    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122044    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.123417 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122047    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122050    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122053    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122056    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122058    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122061    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122063    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122066    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122069    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122072    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122074    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122077    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122079    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122082    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122084    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122087    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122089    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122092    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122095    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122098    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.123934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122100    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122103    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122105    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122109    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122112    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122886    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122894    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122898    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122901    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122903    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122906    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122909    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122912    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122915    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122918    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122921    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122923    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122926    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122929    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.124456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122932    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122934    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122937    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122939    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122942    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122945    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122947    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122950    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122953    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122955    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122958    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122961    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122963    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122966    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122968    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122971    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122973    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122976    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122979    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122982    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.125014 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122984    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122987    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122990    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122992    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122995    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.122998    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123002    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123004    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123007    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123009    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123012    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123014    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123017    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123019    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123022    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123024    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123027    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123030    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123032    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123035    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.125514 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123037    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123039    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123042    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123045    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123048    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123051    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123053    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123056    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123058    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123061    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123063    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123067    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123069    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123072    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123074    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123081    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123086    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123092    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123097    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123101    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.126028 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123105    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123109    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123114    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123118    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123122    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123125    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123127    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123130    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123133    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123136    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123139    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123142    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123144    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123147    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123150    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123152    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123155    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123158    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123161    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.126553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123163    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123166    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123168    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123171    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123173    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123177    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123179    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123183    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123188    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123191    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123195    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123198    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123201    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123689    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123694    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123698    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123700    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123703    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123705    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.127047 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123708    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123711    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123713    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123716    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123719    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123721    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123724    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123727    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123729    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123732    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123735    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123737    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123740    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123742    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123745    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123748    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123750    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123753    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123756    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.127511 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123758    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123761    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123763    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123766    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123771    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123774    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123776    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123779    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123781    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123784    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123789    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123792    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123795    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123797    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123800    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123803    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123805    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123808    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123810    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123813    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.128017 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123815    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123818    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123822    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123826    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123829    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123832    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123835    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123838    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123841    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123843    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123846    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123848    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123851    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123855    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123857    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123860    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123864    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123866    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123869    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.128517 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123871    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123874    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123877    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123880    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123883    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123885    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123888    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123891    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123893    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123896    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123898    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123901    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123904    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123906    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123908    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123911    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123913    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123916    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123918    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123921    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.129015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123923    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123926    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123928    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123931    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123933    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123936    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123939    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123942    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123945    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123949    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123951    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123954    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123957    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123959    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123962    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123966    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123969    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123971    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123974    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123976    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.129504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123979    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.123983    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124084    6695 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124099    6695 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124105    6695 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124110    6695 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124114    6695 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124117    6695 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124121    6695 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124125    6695 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124129    6695 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124132    6695 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124135    6695 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124139    6695 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124142    6695 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124145    6695 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124148    6695 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124151    6695 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124154    6695 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124158    6695 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124162    6695 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124167    6695 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124170    6695 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:10:17.130010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124174    6695 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124177    6695 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124181    6695 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124184    6695 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124189    6695 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124194    6695 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124199    6695 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124203    6695 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124208    6695 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124213    6695 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124217    6695 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124225    6695 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124230    6695 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124233    6695 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124236    6695 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124239    6695 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124243    6695 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124246    6695 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124250    6695 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124253    6695 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124256    6695 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124259    6695 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124263    6695 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124266    6695 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124270    6695 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:10:17.130927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124273    6695 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124276    6695 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124279    6695 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124282    6695 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124286    6695 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124290    6695 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124294    6695 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124297    6695 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124301    6695 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124304    6695 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124307    6695 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124312    6695 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124315    6695 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124318    6695 flags.go:64] FLAG: --help="false"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124321    6695 flags.go:64] FLAG: --hostname-override="ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124324    6695 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124327    6695 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124330    6695 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124333    6695 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124337    6695 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124340    6695 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124342    6695 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124345    6695 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:10:17.131701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124349    6695 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124352    6695 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124355    6695 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124358    6695 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124361    6695 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124364    6695 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124367    6695 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124370    6695 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124372    6695 flags.go:64] FLAG: --lock-file=""
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124375    6695 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124378    6695 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124382    6695 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124387    6695 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124390    6695 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124395    6695 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124398    6695 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124401    6695 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124404    6695 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124407    6695 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124410    6695 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124416    6695 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124420    6695 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124423    6695 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124426    6695 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124430    6695 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:10:17.132276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124433    6695 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124436    6695 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124439    6695 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124442    6695 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124445    6695 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124452    6695 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124455    6695 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124458    6695 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124462    6695 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124465    6695 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124468    6695 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124471    6695 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124474    6695 flags.go:64] FLAG: --port="10250"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124477    6695 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124480    6695 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00121f904ebad0248"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124483    6695 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124486    6695 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124489    6695 flags.go:64] FLAG: --register-node="true"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124492    6695 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124496    6695 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124499    6695 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124503    6695 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124507    6695 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124511    6695 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124514    6695 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124517    6695 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:10:17.132896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124520    6695 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124524    6695 flags.go:64] FLAG: --runonce="false"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124527    6695 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124530    6695 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124533    6695 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124536    6695 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124539    6695 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124542    6695 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124545    6695 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124548    6695 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124551    6695 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124554    6695 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124557    6695 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124560    6695 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124563    6695 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124566    6695 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124569    6695 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124575    6695 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124578    6695 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124581    6695 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124585    6695 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124587    6695 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124591    6695 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124593    6695 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124596    6695 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:10:17.133541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124612    6695 flags.go:64] FLAG: --v="2"
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124617    6695 flags.go:64] FLAG: --version="false"
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124622    6695 flags.go:64] FLAG: --vmodule=""
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124626    6695 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.124630    6695 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124785    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124789    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124793    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124796    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124799    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124802    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124805    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124809    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124811    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124814    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124816    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124819    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124821    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124824    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124827    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.134265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124829    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124832    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124835    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124837    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124847    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124851    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124854    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124857    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124859    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124862    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124865    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124867    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124870    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124872    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124875    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124877    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124880    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124883    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124886    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.134767 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124889    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124892    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124894    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124897    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124900    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124902    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124905    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124907    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124910    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124913    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124915    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124918    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124920    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124923    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124925    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124927    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124930    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124932    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124935    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.135236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124937    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124940    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124942    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124945    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124947    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124950    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124952    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124955    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124957    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124960    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124962    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124965    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124969    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124973    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124976    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124978    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124981    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124983    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124986    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.135788 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124988    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124993    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124995    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.124998    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125001    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125003    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125006    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125008    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125011    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125013    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125016    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125018    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125021    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125024    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125029    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125031    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125034    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125037    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125039    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125042    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.136584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125045    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125047    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125050    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125052    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125055    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125057    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125061    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125065    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125068    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125071    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125074    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125076    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125079    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125083    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.125108    6695 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125146    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.137321 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125150    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125153    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125156    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125159    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125162    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125165    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125167    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125170    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125173    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125175    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125178    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125182    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125185    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125187    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125190    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125193    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125196    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125200    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125204    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125206    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.137853 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125209    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125212    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125214    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125217    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125220    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125223    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125225    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125227    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125230    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125235    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125237    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125240    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125244    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125248    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125252    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125256    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125260    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125264    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125267    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.138338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125272    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125276    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125280    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125284    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125288    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125291    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125294    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125297    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125299    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125302    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125305    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125307    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125310    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125313    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125315    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125318    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125321    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125324    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125327    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.138926 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125330    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125333    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125335    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125339    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125342    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125344    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125347    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125349    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125352    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125355    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125357    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125360    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125362    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125365    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125367    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125370    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125373    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125377    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125379    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.139458 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125382    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125384    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125387    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125389    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125392    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125395    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125397    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125400    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125402    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125405    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125408    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125410    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125415    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125418    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125421    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125424    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125428    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125431    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125433    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.140121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125436    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125438    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125441    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125443    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125447    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125450    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125452    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125455    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.125457    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.129923    6695 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.129940    6695 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.129960    6695 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.129965    6695 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130036    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130041    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130044    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130047    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130051    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130054    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130057    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.140698 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130059    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130062    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130064    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130236    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130244    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130250    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130258    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130264    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130272    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130279    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130284    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130349    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130357    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130360    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130365    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130369    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130383    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130388    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130394    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.141191 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130408    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130416    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130423    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130482    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130488    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130493    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130498    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130502    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130507    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130516    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130521    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130525    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130530    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130534    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130538    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130543    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130547    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130552    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130557    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130561    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.141670 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130565    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130569    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130578    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130583    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130588    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130592    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130599    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130628    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130634    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130639    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130644    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130648    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130653    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130657    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130662    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130672    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130704    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130710    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130714    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.142153 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130718    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130722    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130727    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130731    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130736    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130740    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130745    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130750    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130759    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130763    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130767    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130771    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130776    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130780    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130787    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130793    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130799    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130804    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130809    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.142667 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130813    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130822    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130826    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130830    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130835    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130840    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130844    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130848    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130852    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130857    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130861    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130891    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130896    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130900    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130911    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130916    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130920    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130924    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130929    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130934    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.143151 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130938    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.130943    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.131004    6695 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131563    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131577    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131581    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131584    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131587    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131590    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131593    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131596    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131612    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131617    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131621    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131623    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131626    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.143679 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131629    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131632    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131635    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131637    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131640    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131643    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131646    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131649    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131652    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131655    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131658    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131660    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131663    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131666    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131670    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131676    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131679    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131683    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131685    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131689    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.144071 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131694    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131698    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131700    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131703    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131707    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131709    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131713    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131716    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131719    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131721    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131724    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131727    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131730    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131732    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131735    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131737    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131740    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131742    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131745    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.144542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131747    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131750    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131752    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131755    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131757    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131760    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131763    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131765    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131768    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131771    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131774    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131778    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131781    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131784    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131787    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131790    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131792    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131795    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131798    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.145016 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131801    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131803    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131806    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131809    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131812    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131814    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131816    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131819    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131822    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131824    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131827    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131830    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131833    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131836    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131846    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131849    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131852    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131854    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131857    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131859    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.145494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131862    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131865    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131868    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131870    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131873    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131876    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131878    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131881    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131883    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131886    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131888    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131891    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131893    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131896    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131899    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.131927    6695 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:10:17.145999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131963    6695 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131968    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131971    6695 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131974    6695 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131978    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131981    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131984    6695 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131987    6695 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131990    6695 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131993    6695 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.131996    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132000    6695 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132003    6695 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132005    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132009    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132012    6695 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132014    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132017    6695 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132019    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:10:17.146392 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132022    6695 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132025    6695 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132027    6695 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132030    6695 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132032    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132035    6695 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132038    6695 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132040    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132043    6695 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132045    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132048    6695 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132051    6695 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132054    6695 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132056    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132059    6695 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132061    6695 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132064    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132066    6695 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132069    6695 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132072    6695 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:10:17.146857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132074    6695 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132077    6695 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132080    6695 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132086    6695 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132089    6695 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132092    6695 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132095    6695 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132098    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132101    6695 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132103    6695 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132106    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132109    6695 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132112    6695 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132114    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132117    6695 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132119    6695 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132122    6695 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132124    6695 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132127    6695 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132130    6695 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:10:17.147356 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132132    6695 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132135    6695 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132138    6695 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132140    6695 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132143    6695 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132146    6695 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132148    6695 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132151    6695 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132154    6695 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132156    6695 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132160    6695 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132163    6695 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132166    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132171    6695 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132174    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132179    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132182    6695 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132185    6695 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132188    6695 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:10:17.147943 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132191    6695 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132194    6695 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132196    6695 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132199    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132202    6695 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132205    6695 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132207    6695 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132210    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132212    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132215    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132217    6695 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132220    6695 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132222    6695 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132226    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132228    6695 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132231    6695 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132233    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132236    6695 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132238    6695 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132241    6695 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:10:17.148422 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132244    6695 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132246    6695 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132249    6695 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132256    6695 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132259    6695 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132264    6695 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132267    6695 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:17.132270    6695 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.132346    6695 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.133410    6695 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.133462    6695 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.133664    6695 server.go:1014] "Starting client certificate rotation"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.133796    6695 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.133850    6695 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:04:06 +0000 UTC" deadline="2028-08-11 21:18:18.596100308 +0000 UTC"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.133877    6695 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15624h8m1.462225123s"
Oct 30 21:10:17.149170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.134092    6695 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:10:17.149797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.135163    6695 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:10:17.149797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.137104    6695 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:10:17.149797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.139194    6695 log.go:25] "Validated CRI v1 image API"
Oct 30 21:10:17.149797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.139946    6695 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:10:17.149797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.142212    6695 fs.go:135] Filesystem UUIDs: map[52f30e7f-b0cf-465b-9b0c-f4ee43bf6849:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c92ad68e-9432-4cb3-a3e2-9d2e0ba954db:/dev/nvme0n1p3]
Oct 30 21:10:17.150057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.142234    6695 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83/userdata/shm major:0 minor:446 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4979cc278591ce7eded65e70bfdd43754320c03d4fbdad853dec669bb87b491a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4979cc278591ce7eded65e70bfdd43754320c03d4fbdad853dec669bb87b491a/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba/userdata/shm major:0 minor:441 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561/userdata/shm major:0 minor:444 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volumes/kubernetes.io~projected/kube-api-access-274cp:{mountpoint:/var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volumes/kubernetes.io~projected/kube-api-access-274cp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~projected/kube-api-access-tkjxk:{mountpoint:/var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~projected/kube-api-access-tkjxk major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19448ad0-e5a8-4760-b4c5-e8410e14af1b/volumes/kubernetes.io~projected/kube-api-access-2nkxm:{mountpoint:/var/lib/kubelet/pods/19448ad0-e5a8-4760-b4c5-e8410e14af1b/volumes/kubernetes.io~projected/kube-api-access-2nkxm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327/volumes/kubernetes.io~projected/kube-api-access-trphj:{mountpoint:/var/lib/kubelet/pods/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327/volumes/kubernetes.io~projected/kube-api-access-trphj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca7df3d-5d59-436f-a381-a0224e48272f/volumes/kubernetes.io~projected/kube-api-access-s9fl2:{mountpoint:/var/lib/kubelet/pods/2ca7df3d-5d59-436f-a381-a0224e48272f/volumes/kubernetes.io~projected/kube-api-access-s9fl2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca7df3d-5d59-436f-a381-a0224e48272f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/2ca7df3d-5d59-436f-a381-a0224e48272f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~projected/kube-api-access-s697g:{mountpoint:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~projected/kube-api-access-s697g major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34a0ac7b-1822-4157-9438-d76e6783177e/volumes/kubernetes.io~projected/kube-api-access-5hgkt:{mountpoint:/var/lib/kubelet/pods/34a0ac7b-1822-4157-9438-d76e6783177e/volumes/kubernetes.io~projected/kube-api-access-5hgkt major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34a0ac7b-1822-4157-9438-d76e6783177e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/34a0ac7b-1822-4157-9438-d76e6783177e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38c2f0bf-d12e-497e-b64b-220761a147a6/volumes/kubernetes.io~projected/kube-api-access-q4r9r:{mountpoint:/var/lib/kubelet/pods/38c2f0bf-d12e-497e-b64b-220761a147a6/volumes/kubernetes.io~projected/kube-api-access-q4r9r major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c61a330-f062-4cbf-9794-3f7973e3826d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3c61a330-f062-4cbf-9794-3f7973e3826d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67203193-932d-445f-b14d-457040dcee8e/volumes/kubernetes.io~projected/kube-api-access-c4ss9:{mountpoint:/var/lib/kubelet/pods/67203193-932d-445f-b14d-457040dcee8e/volumes/kubernetes.io~projected/kube-api-access-c4ss9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~projected/kube-api-access-pb96b:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~projected/kube-api-access-pb96b major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~projected/kube-api-access-rj6p4:{mountpoint:/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~projected/kube-api-access-rj6p4 major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7baeebaf-f8c0-4643-b921-fff75980046a/volumes/kubernetes.io~projected/kube-api-access-f6n7c:{mountpoint:/var/lib/kubelet/pods/7baeebaf-f8c0-4643-b921-fff75980046a/volumes/kubernetes.io~projected/kube-api-access-f6n7c major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7baeebaf-f8c0-4643-b921-fff75980046a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7baeebaf-f8c0-4643-b921-fff75980046a/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~projected/kube-api-access-j7twc:{mountpoint:/var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~projected/kube-api-access-j7twc major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88e4b3aa-5e27-4609-ae83-e74875a7e8da/volumes/kubernetes.io~projected/kube-api-access-cwhqr:{mountpoint:/var/lib/kubelet/pods/88e4b3aa-5e27-4609-ae83-e74875a7e8da/volumes/kubernetes.io~projected/kube-api-access-cwhqr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e/volumes/kubernetes.io~projected/kube-api-access-khxvk:{mountpoint:/var/lib/kubelet/pods/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e/volumes/kubernetes.io~projected/kube-api-access-khxvk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/900d3f13-ab1c-478e-9851-a70264d92012/volumes/kubernetes.io~projected/kube-api-access-cs5f5:{mountpoint:/var/lib/kubelet/pods/900d3f13-ab1c-478e-9851-a70264d92012/volumes/kubernetes.io~projected/kube-api-access-cs5f5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/900d3f13-ab1c-478e-9851-a70264d92012/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/900d3f13-ab1c-478e-9851-a70264d92012/volumes/kubernetes.io~secret/metrics-certs major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b394b4-d048-4139-a5eb-c32918e048d2/volumes/kubernetes.io~projected/kube-api-access-rcxwn:{mountpoint:/var/lib/kubelet/pods/93b394b4-d048-4139-a5eb-c32918e048d2/volumes/kubernetes.io~projected/kube-api-access-rcxwn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a33b283-c453-4742-9142-173f02cadeab/volumes/kubernetes.io~projected/kube-api-access-4fcdb:{mountpoint:/var/lib/kubelet/pods/9a33b283-c453-4742-9142-173f02cadeab/volumes/kubernetes.io~projected/kube-api-access-4fcdb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~projected/kube-api-access-jgz58:{mountpoint:/var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~projected/kube-api-access-jgz58 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~projected/kube-api-access-9br6h:{mountpoint:/var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~projected/kube-api-access-9br6h major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/alertmanager/9 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~projected/kube-api-access-ntt6d:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~projected/kube-api-access-ntt6d major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~projected/tls-assets major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/config-volume major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/web-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c567c895-2164-4fdc-a4b4-4df9af394efc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c567c895-2164-4fdc-a4b4-4df9af394efc/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2a3db9d5aa1a44cedb317f10030b4b52870c13b768b2d904051f45929ca0fadc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2d14ee7598ab44210d73bbd5db31a94278a658fa384c620e0451bb1ecb9890eb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d1e70d4aed812dce4d30cf39f82d5fefa3a08b1b26b9411a6bbe493ac01d67f6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9c13e564c1a6e4c3a7a3e5b2946e2e69a68133d7861285c3936e7e728d47f165/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7f1330c1dc6a2be34ce9a4c541fb5085b0d878a48f64b96c2c16e797ab395376/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d51dbf285be6a795a1a4002415983b16c2ce67706c421455c603826abecec234/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8843aba14fb1379f1c2c85cf4447e8a2bee5ff25c256771ce01b371f3ab36cf1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1a68c9085f4db6accfef4815547987280a817fc7fc9a1ce0c14645a27477dbd6/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f4f675e67e99b66377f71570609f1b3290a0d7e45699649c0f4f1cee169f0d0d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/d7703a183e9327811156ab7dc1b9e9b4ae280629bc0904301c886486bc6c206a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2036f289eae420a3f6a0c73d32e8c328751499c04424f415e9f20d6ebd84c2b6/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/377b88ac83ccc9890e3d5cb17c60e619fffb2ee6d903528c3ce514b83260c4e2/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1691e1c7f81a99735dafdad11725ed4c639c3df67bb9502ba23a22c7778b224c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d070cf42965383d00ea9aa83274364e2b229f0206387e29d234c8a1d9d5c2eb8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1f4fdc0185101218a968051d66ec5d84fe916c1bf64d035a7f3302396081210c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b0d6cd7c8d34fb42e3f722b8d0fb31524acd2c862d7383de561e3f2c1ea46512/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ce85404a1afe51f3c30a98aceb15910a12e074e1be3d615a85594c746138bdaa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/3d516568875c2e8c8fb37a3017244858c92394a78bedf27f3b759bc3b51d5d73/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/cb250ba3f8f75d5fabb4cd06acd54197d836ed3f2591619562d6124e150941ec/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2902c57e94c3d8cd3f2c72476fa318205b964b4f9b3cdb5a5c207fcc906c7f62/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9da7aa85d5e459c4fd5355bde1b44d4bca9cedaac700a87d01db73245fb75ea1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/13e0062b8ed257b8cd4147ebf9e7a4023c797cd6d15f235e6372a7ff8172a249/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2b923bd91e6e5bdd3bf505e4afead583c6e61b585f26c0c5d66289895a6e4401/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e09b20783294871984b73136883f846be1817430d9c468088cb3c48861ac03f8/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/b57d57d0df28eef3154bf27b00721682806b0a8add83965ce059571fd6732f63/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/78ca72dc4b39552f01e961bb58f48d6b660e1d2077905c3edddbdbf20733f179/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/c82330b678cf29b29e2c8d27f89051f89ccd3296a6dd92715dbdecdddaaa892c/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/3e5d1832edb51d503e5418d5f15b65fdffc8bedf2b6606c233ecad800d3a16f1/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d1959c3a2bf34f60c3aa2a40cff9e771098dde6cac6dcb351b7ebbe1e85d7aef/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/b39c073f6e0c04c2005eafb5705b2d793ca467be083d6fa072fef6f192b04367/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/c3e37e09b0cb33d89bcad030a987a3f253173e8cbb6d8b66a4b8cbf243768c97/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/90ce2c33405045c697e68ccb6bc203fcf49e9f9dd44edb8caf92b2dddda89ec8/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2f7f39cdff6592654466dae8bce45028371705c814a20d2cfd344b537474151a/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/74044c3a2ce59b7151412afe3541f0a8011bbed34a980cf06dc7a763a7c17681/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/9e40124359504ee1c7ebd6d225e3121ddf7892d49e847dba89cefdefc3a9b745/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d22591783d85950b9aeeb12f553917486a0a9c988f8a13358bbfae15c5353832/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/249e18bda6e3e9cf4e04cd2151fd543153ab5aae784a4a0f54d23662dbc87a17/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/6647d6c87c4c965867a1aec87b634fa07ebec4c4066cf27d88ae0dd17c7c4234/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/55facff4cbf7b59707677b5dd62be72e4e4312469826069ee383f3942fb1e5cb/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/dcc1b852350262dab6e4aef5d4c2379595a9b88bc00e471f5f2294f2579bb2c6/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/d1cb45f1b3484274cb5cc41fff1f814047f9bc9a92c3ef75ddef0fba4b869bae/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/a3a5e54db26ff5520bee73d21767eee698ed966b523f48756884a1dffd33e7d0/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/4a9b0b41984d27f32f814d7f7214062faebfd5ab422e3ab7b426e84e42d442b8/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c174901a22fc08c4ec9f554262da4122d9dce5bfe9882ec8395f96a60adf03a7/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/c1feaebe291add63870339c28dd5b7f80455b17235f3f3c65b3d9c419a50395b/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/00cf2abc96685ee89a519ae2eb85327366f655b7a4750230f3472ad9caf3541f/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/7dee5ae54a45f1d15def03c9d75629e0aa34be027a58c43eb73aa154e4742700/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/39f26b51640926f953eb9980de3551e9dca571fe5bd8df27355db0ba56d24034/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/9fdea7ef9ec6976dc08c6a29a35d0e10f878aa4b37509f26f2393a3b45343fdc/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/41342e9d43f6458f19eb2f4fccc1286f51fdd8967e16989ce708d7389a5c1753/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/61b6437c9cf4655434de2a76b794f45d06c89988d381e7971e9b23a987503b31/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/90893ad7def7d7b30ecb2ef44990168c6075bfdbe3bda096d162b4f92d7c34fe/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/15087b539bdca04e39c4390f997c2cff17283f16daf7888361c5fcd048dbf025/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/420d4cabcfb0033a6c38bd722656a2911ff7e3e7b0f0b4d7826d73dfa23eedda/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/473ec91df5009b9464624a1b0ba86194bf7966ac1822a2d0db9cd84ecae89425/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/f259cd413a4a351de84b86d55f73534ff79e112f5460dfbb8e9b1edb68d2b58b/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/2ebb869106d2f1647350708c25613e505d98f35b8de2b40e4a3b00312b431632/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/6ca05040c2c31edd0da181e42b3b8b61d922567e504d654a52b8eaf6ff52a6a3/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/20c3870f1781ddb887ae49621513f47a764c344ff841e8559e8fe7d88fb93e0b/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/89cd1fbc71daef04cbfb68780a166807437d9eced14de10a580c18fddcd17b12/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/eafeb13c152f0a576bed899ce354d6964310e28fd532c137bde4e8b08d46f3c6/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/e28f47d3666415136dd4dcfefda701f84560f2ff87df222121ecd9697a966724/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/434b162c2742c276615b81025966e2d12db19e4b60e2c935906b0c2ce714c318/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/b014991278a43b672f93e27e723eb24d60c84e029a63b5ad0debf1e26a21436f/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/48d4fcd7c9e17f7ca459061abaecd23fa6acfc2447e7b3ea5d4e85d57ef544b8/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/2ca56c27b510e18cbbbcc2020796e86a2f29fccc01bfedbf941f872666f9cf49/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/590c5f302f8e6b7ee5a463aa9881ffe41cb5fbc4b47fa7302c1f651aca769cdd/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/1d17baad260483381f6e505ab7fa2da06fd4db19637b44ac596cab8dadc4646a/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1bfb6bdb1b23558523abd164c7a984048d82990ab51693c70dc094a56efa1144/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d17f177f41425c3c7de0e4a46da2fc5a2b56623c6888ed39681f037b26273184/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7d2a6c7e8bcc623d7958040555b6e09d5880b81d666a075f4725a58bb8825587/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3af3ba0fd847e0f901d325254510e2d3fad70259c2818812296fdccec5d490dd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/eaf0863b71e887bb89f863b9b7eb9c37555cd9096655bc90953131395257eb75/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6c4a7bfc158dcc92c6c9bdc656f8d8c98eaa6f5231f91e047e2f0bfc99541a3f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/fb30c231a45ca83943dd6e494281369640f70d697eea9d06f8023cb395a98ac0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/86895aa8a66930049821f2dc47d7cb9b8dff2b106750503d1b8f2afdd6cb56bd/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2bfed45e10a11e17bc699504e9c5363c8723686b9bd8b06d3100cb7773f4537c/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/48617d3b707e96236f40acd51979d18afeee017401369bfec516f5e28492711d/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:10:17.152200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.151737    6695 manager.go:217] Machine: {Timestamp:2026-10-30 21:10:17.151243597 +0000 UTC m=+0.073701886 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102286 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20a09fc4cf9a43a1abb76d4471cd06 SystemUUID:ec20a09f-c4cf-9a43-a1ab-b76d4471cd06 BootID:610b313d-715e-4f6f-9f3a-f2c9e77b6bcf Filesystems:[{Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~projected/kube-api-access-pb96b DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a341cd7325ed9c1cbd210f7c5a2e5fb2351e3823e278e82398bed4a6ded3bf58/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93b394b4-d048-4139-a5eb-c32918e048d2/volumes/kubernetes.io~projected/kube-api-access-rcxwn DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/900d3f13-ab1c-478e-9851-a70264d92012/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~projected/kube-api-access-rj6p4 DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c7e58126cd83f3a2c86e4d03d65df90cec0be18d60cbb3699e221141c044614/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volumes/kubernetes.io~projected/kube-api-access-274cp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1858afedd4cf98b151ea1971055bdc6b840d6996c23adb418cfaf068379a739a/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34a0ac7b-1822-4157-9438-d76e6783177e/volumes/kubernetes.io~projected/kube-api-access-5hgkt DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:274 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea6011c686031d2c2852ec80bb70f7daa0a7db18110245c3732cced3bd28a6f4/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/438c9c0e340c7dd21f1207b2832ee39025ad3a29463d81457e0c601a0b5c15ca/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e/volumes/kubernetes.io~projected/kube-api-access-khxvk DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63832eb8e46178be81c0a9e2e174228e91de5da85b604b7f8536fa02942c4bf2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~projected/kube-api-access-ntt6d DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad4994343823a04a6c3b28e3e6bcf0361c95656be3c278678212f00f28e50d8b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~projected/kube-api-access-9br6h DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad9c3e1ae2faa7591744e480206d777cee0e43cc1686d3e7e7cc4581ca4ef561/userdata/shm DeviceMajor:0 DeviceMinor:444 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a33b283-c453-4742-9142-173f02cadeab/volumes/kubernetes.io~projected/kube-api-access-4fcdb DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~projected/kube-api-access-s697g DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34a0ac7b-1822-4157-9438-d76e6783177e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7686f5c0af1a73971d7641a3c31790506e09fea82fed23685f904244e20c1d1a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7baeebaf-f8c0-4643-b921-fff75980046a/volumes/kubernetes.io~projected/kube-api-access-f6n7c DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89595e1063726e80fe7bf9783e1cc872375c65a8bf58ecfa778d74ff078b197f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~projected/kube-api-access-jgz58 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9bb94f82fe7b6b339a557a41ff0ee70970264e43d1b71f7397e6934945adc92/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38c2f0bf-d12e-497e-b64b-220761a147a6/volumes/kubernetes.io~projected/kube-api-access-q4r9r DeviceMajor:0 DeviceMinor:443 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23ff0dcfd74b049fe605782fb365cc80d1270f176ca1323a7bedd81c634434e2/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5b434c4a37277dd50a6110e072a5c7385a546dd82dc065fed2816b693396bd2/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:272 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/054fd709-2d73-4ce0-ab94-0df9fbd1921a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fc3505e4557ded3caf4425351bb96307044c511faeeacb7718987f7d9439c02/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327/volumes/kubernetes.io~projected/kube-api-access-trphj DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67790f00-7ce5-4f74-ae11-893cf8e217ad/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7775f2b05e2c034a19e91c1959724f90ae957f0ed4356907c0f163ff392c2814/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7baeebaf-f8c0-4643-b921-fff75980046a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3c61a330-f062-4cbf-9794-3f7973e3826d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:440 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14ba1facb17ec7e4ad1a81ba54ff49165b798cb7b1179650d7bac133db9c2a83/userdata/shm DeviceMajor:0 DeviceMinor:446 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e92c7f58d822bf6b9dcc17892094fbca866215040b70a133c8c956845120b99/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85f8f57fd6bfea159cc2f884146406430590a5c4fadb159e6fd298af8d32c0ba/userdata/shm DeviceMajor:0 DeviceMinor:441 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:268 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19448ad0-e5a8-4760-b4c5-e8410e14af1b/volumes/kubernetes.io~projected/kube-api-access-2nkxm DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88e4b3aa-5e27-4609-ae83-e74875a7e8da/volumes/kubernetes.io~projected/kube-api-access-cwhqr DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/67203193-932d-445f-b14d-457040dcee8e/volumes/kubernetes.io~projected/kube-api-access-c4ss9 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4979cc278591ce7eded65e70bfdd43754320c03d4fbdad853dec669bb87b491a/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5962d044e9fc9ee70c900b49db2c7fc00137c2e7b8f7bbb38dba7a7a00adc547/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/900d3f13-ab1c-478e-9851-a70264d92012/volumes/kubernetes.io~projected/kube-api-access-cs5f5 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca7df3d-5d59-436f-a381-a0224e48272f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d13165-1255-4e98-b0c8-a84b78043c93/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c567c895-2164-4fdc-a4b4-4df9af394efc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2eed9c68-fae8-40b3-9bee-499e396d96e5/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/volumes/kubernetes.io~projected/kube-api-access-tkjxk DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/volumes/kubernetes.io~projected/kube-api-access-j7twc DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2806653-74fd-4d37-a4ba-459232f41e74/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca7df3d-5d59-436f-a381-a0224e48272f/volumes/kubernetes.io~projected/kube-api-access-s9fl2 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14ba1facb17ec7e MacAddress:3e:d2:b1:1c:bf:7b Speed:10000 Mtu:8901} {Name:23ff0dcfd74b049 MacAddress:a2:a2:59:3d:c5:fa Speed:10000 Mtu:8901} {Name:438c9c0e340c7dd MacAddress:8a:e2:fc:8a:5c:9a Speed:10000 Mtu:8901} {Name:5962d044e9fc9ee MacAddress:7a:e8:81:cc:5c:c0 Speed:10000 Mtu:8901} {Name:6fc3505e4557ded MacAddress:fe:5b:13:c9:ab:40 Speed:10000 Mtu:8901} {Name:7775f2b05e2c034 MacAddress:26:66:39:86:e6:a9 Speed:10000 Mtu:8901} {Name:7c7e58126cd83f3 MacAddress:f6:40:b4:f5:f9:eb Speed:10000 Mtu:8901} {Name:85f8f57fd6bfea1 MacAddress:e2:4c:fc:fd:a7:32 Speed:10000 Mtu:8901} {Name:89595e1063726e8 MacAddress:06:24:17:d2:bd:3a Speed:10000 Mtu:8901} {Name:8bc6e9824a7374f MacAddress:7a:af:36:8e:9b:5e Speed:10000 Mtu:8901} {Name:a341cd7325ed9c1 MacAddress:c2:0e:39:68:24:da Speed:10000 Mtu:8901} {Name:ad9c3e1ae2faa75 MacAddress:9a:2b:9b:b6:ad:27 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:66:46:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:0f:05:67:7a:6b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:66:46:6d Speed:0 Mtu:9001} {Name:f87ffc7822d3ac9 MacAddress:f6:b7:ea:e5:a3:5d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:da:34:c9:55:a2:71 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:4f:99:40:2a:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:10:17.152434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152205    6695 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:10:17.152434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152258    6695 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:10:17.152631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152577    6695 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:10:17.152800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152626    6695 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:10:17.152892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152805    6695 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:10:17.152892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152818    6695 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:10:17.152892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152832    6695 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:10:17.152892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.152870    6695 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:10:17.153079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153007    6695 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:10:17.153240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153227    6695 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:10:17.153367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153357    6695 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:10:17.153428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153406    6695 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:10:17.153428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153425    6695 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:10:17.153513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153441    6695 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:10:17.153513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153456    6695 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:10:17.153513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.153467    6695 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:10:17.154212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.154194    6695 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:10:17.154212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.154210    6695 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:10:17.154743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.154720    6695 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:10:17.157547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.157531    6695 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:10:17.157547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.157545    6695 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:10:17.157770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.157660    6695 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:10:17.157882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.157869    6695 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:10:17.157912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.157900    6695 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:10:17.158028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158018    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158038    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158048    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158056    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158063    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158069    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158074    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158080    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158087    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158093    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158105    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:10:17.158113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158114    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:10:17.158397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158142    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:10:17.158397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.158148    6695 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:10:17.161671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.161659    6695 server.go:1263] "Started kubelet"
Oct 30 21:10:17.162320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.161989    6695 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:10:17.162342 ip-10-0-142-232 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:10:17.163864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.162421    6695 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:10:17.164029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.164015    6695 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:10:17.167462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.165066    6695 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:10:17.168247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.168228    6695 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:10:17.170378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170359    6695 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:10:17.170462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170381    6695 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:10:17.170462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170428    6695 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:04:07 +0000 UTC" deadline="2028-05-01 14:45:17.199527204 +0000 UTC"
Oct 30 21:10:17.170462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170448    6695 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13169h35m0.029082676s"
Oct 30 21:10:17.170650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170475    6695 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:10:17.170650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170493    6695 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:10:17.170650 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.170499    6695 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-232.ec2.internal\" not found"
Oct 30 21:10:17.170650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.170581    6695 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:10:17.173759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173716    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a33b283-c453-4742-9142-173f02cadeab" volumeName="kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:10:17.173825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173763    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2806653-74fd-4d37-a4ba-459232f41e74" volumeName="kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-etc-tuned" seLinuxMountContext=""
Oct 30 21:10:17.173825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173786    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c05ea68-c44a-4fbc-bda8-e6cb9ea15327" volumeName="kubernetes.io/empty-dir/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-data-volume" seLinuxMountContext=""
Oct 30 21:10:17.173825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173810    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173826    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173841    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d13165-1255-4e98-b0c8-a84b78043c93" volumeName="kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173853    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c567c895-2164-4fdc-a4b4-4df9af394efc" volumeName="kubernetes.io/configmap/c567c895-2164-4fdc-a4b4-4df9af394efc-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173865    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="054fd709-2d73-4ce0-ab94-0df9fbd1921a" volumeName="kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-env-overrides" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173880    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173894    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173908    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out" seLinuxMountContext=""
Oct 30 21:10:17.173925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173921    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173935    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9f385c-ce7e-42d9-a79c-2ef0976f44f5" volumeName="kubernetes.io/empty-dir/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-volume-directive-shadow" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173948    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/configmap/67790f00-7ce5-4f74-ae11-893cf8e217ad-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173965    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" volumeName="kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173978    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d13165-1255-4e98-b0c8-a84b78043c93" volumeName="kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.173990    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca7df3d-5d59-436f-a381-a0224e48272f" volumeName="kubernetes.io/empty-dir/2ca7df3d-5d59-436f-a381-a0224e48272f-tmp-dir" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174011    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/projected/2eed9c68-fae8-40b3-9bee-499e396d96e5-kube-api-access-s697g" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174025    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174038    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38c2f0bf-d12e-497e-b64b-220761a147a6" volumeName="kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174057    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67203193-932d-445f-b14d-457040dcee8e" volumeName="kubernetes.io/configmap/67203193-932d-445f-b14d-457040dcee8e-serviceca" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174069    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d13165-1255-4e98-b0c8-a84b78043c93" volumeName="kubernetes.io/configmap/b3d13165-1255-4e98-b0c8-a84b78043c93-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174081    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174108    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174142    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c05ea68-c44a-4fbc-bda8-e6cb9ea15327" volumeName="kubernetes.io/configmap/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174158    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca7df3d-5d59-436f-a381-a0224e48272f" volumeName="kubernetes.io/configmap/2ca7df3d-5d59-436f-a381-a0224e48272f-config-volume" seLinuxMountContext=""
Oct 30 21:10:17.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174182    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34a0ac7b-1822-4157-9438-d76e6783177e" volumeName="kubernetes.io/secret/34a0ac7b-1822-4157-9438-d76e6783177e-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174215    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" volumeName="kubernetes.io/projected/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-kube-api-access-j7twc" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174248    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174268    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="054fd709-2d73-4ce0-ab94-0df9fbd1921a" volumeName="kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-config" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174290    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9f385c-ce7e-42d9-a79c-2ef0976f44f5" volumeName="kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174326    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19448ad0-e5a8-4760-b4c5-e8410e14af1b" volumeName="kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174363    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca7df3d-5d59-436f-a381-a0224e48272f" volumeName="kubernetes.io/secret/2ca7df3d-5d59-436f-a381-a0224e48272f-metrics-tls" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174380    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174393    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174415    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34a0ac7b-1822-4157-9438-d76e6783177e" volumeName="kubernetes.io/projected/34a0ac7b-1822-4157-9438-d76e6783177e-kube-api-access-5hgkt" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174428    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19448ad0-e5a8-4760-b4c5-e8410e14af1b" volumeName="kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174443    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7baeebaf-f8c0-4643-b921-fff75980046a" volumeName="kubernetes.io/projected/7baeebaf-f8c0-4643-b921-fff75980046a-kube-api-access-f6n7c" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174478    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/empty-dir/67790f00-7ce5-4f74-ae11-893cf8e217ad-tmp" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174499    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" volumeName="kubernetes.io/empty-dir/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-textfile" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174531    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" volumeName="kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-tls" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174422    6695 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174553    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="900d3f13-ab1c-478e-9851-a70264d92012" volumeName="kubernetes.io/projected/900d3f13-ab1c-478e-9851-a70264d92012-kube-api-access-cs5f5" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174567    6695 factory.go:55] Registering systemd factory
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174580    6695 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174616    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a33b283-c453-4742-9142-173f02cadeab" volumeName="kubernetes.io/projected/9a33b283-c453-4742-9142-173f02cadeab-kube-api-access-4fcdb" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174636    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d13165-1255-4e98-b0c8-a84b78043c93" volumeName="kubernetes.io/projected/b3d13165-1255-4e98-b0c8-a84b78043c93-kube-api-access-9br6h" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174662    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9f385c-ce7e-42d9-a79c-2ef0976f44f5" volumeName="kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174674    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf6607a-ab80-47b6-ad1c-e33939b8dc1e" volumeName="kubernetes.io/projected/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kube-api-access-khxvk" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174687    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174713    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174739    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c567c895-2164-4fdc-a4b4-4df9af394efc" volumeName="kubernetes.io/secret/c567c895-2164-4fdc-a4b4-4df9af394efc-agent-certs" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174749    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67203193-932d-445f-b14d-457040dcee8e" volumeName="kubernetes.io/projected/67203193-932d-445f-b14d-457040dcee8e-kube-api-access-c4ss9" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174772    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/projected/67790f00-7ce5-4f74-ae11-893cf8e217ad-kube-api-access-pb96b" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174786    6695 factory.go:153] Registering CRI-O factory
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174801    6695 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174821    6695 factory.go:103] Registering Raw factory
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174837    6695 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174788    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174874    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174903    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config" seLinuxMountContext=""
Oct 30 21:10:17.174901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174917    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174929    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a33b283-c453-4742-9142-173f02cadeab" volumeName="kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174976    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2806653-74fd-4d37-a4ba-459232f41e74" volumeName="kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-tmp" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174989    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2806653-74fd-4d37-a4ba-459232f41e74" volumeName="kubernetes.io/projected/a2806653-74fd-4d37-a4ba-459232f41e74-kube-api-access-jgz58" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.174999    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9f385c-ce7e-42d9-a79c-2ef0976f44f5" volumeName="kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175054    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9f385c-ce7e-42d9-a79c-2ef0976f44f5" volumeName="kubernetes.io/projected/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-api-access-tkjxk" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175080    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175097    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19448ad0-e5a8-4760-b4c5-e8410e14af1b" volumeName="kubernetes.io/projected/19448ad0-e5a8-4760-b4c5-e8410e14af1b-kube-api-access-2nkxm" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175160    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175179    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175213    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="900d3f13-ab1c-478e-9851-a70264d92012" volumeName="kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175231    6695 manager.go:319] Starting recovery of all containers
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175239    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175251    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175262    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19448ad0-e5a8-4760-b4c5-e8410e14af1b" volumeName="kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175294    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c05ea68-c44a-4fbc-bda8-e6cb9ea15327" volumeName="kubernetes.io/secret/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175318    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175335    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-client-ca-bundle" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175348    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175381    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175393    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175437    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7baeebaf-f8c0-4643-b921-fff75980046a" volumeName="kubernetes.io/secret/7baeebaf-f8c0-4643-b921-fff75980046a-cert" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175450    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" volumeName="kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175460    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88e4b3aa-5e27-4609-ae83-e74875a7e8da" volumeName="kubernetes.io/projected/88e4b3aa-5e27-4609-ae83-e74875a7e8da-kube-api-access-cwhqr" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175483    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b394b4-d048-4139-a5eb-c32918e048d2" volumeName="kubernetes.io/projected/93b394b4-d048-4139-a5eb-c32918e048d2-kube-api-access-rcxwn" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175496    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" volumeName="kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175510    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="054fd709-2d73-4ce0-ab94-0df9fbd1921a" volumeName="kubernetes.io/secret/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175541    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c05ea68-c44a-4fbc-bda8-e6cb9ea15327" volumeName="kubernetes.io/projected/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-api-access-trphj" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175554    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2eed9c68-fae8-40b3-9bee-499e396d96e5" volumeName="kubernetes.io/empty-dir/2eed9c68-fae8-40b3-9bee-499e396d96e5-audit-log" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175567    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175588    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67790f00-7ce5-4f74-ae11-893cf8e217ad" volumeName="kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-grpc-tls" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175613    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71a6179c-e8ae-4870-a41e-29e65a6373e1" volumeName="kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175625    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" volumeName="kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175647    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88e4b3aa-5e27-4609-ae83-e74875a7e8da" volumeName="kubernetes.io/configmap/88e4b3aa-5e27-4609-ae83-e74875a7e8da-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175749    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="054fd709-2d73-4ce0-ab94-0df9fbd1921a" volumeName="kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175772    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="054fd709-2d73-4ce0-ab94-0df9fbd1921a" volumeName="kubernetes.io/projected/054fd709-2d73-4ce0-ab94-0df9fbd1921a-kube-api-access-274cp" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175786    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d9f385c-ce7e-42d9-a79c-2ef0976f44f5" volumeName="kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175798    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca7df3d-5d59-436f-a381-a0224e48272f" volumeName="kubernetes.io/projected/2ca7df3d-5d59-436f-a381-a0224e48272f-kube-api-access-s9fl2" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175810    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c61a330-f062-4cbf-9794-3f7973e3826d" volumeName="kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175850    6695 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b394b4-d048-4139-a5eb-c32918e048d2" volumeName="kubernetes.io/empty-dir/93b394b4-d048-4139-a5eb-c32918e048d2-tmp-dir" seLinuxMountContext=""
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175866    6695 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:10:17.176487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.175873    6695 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:10:17.203876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.203770    6695 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:10:17.204594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.204571    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:10:17.205342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.205322    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:10:17.205722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.205465    6695 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:10:17.205817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.205726    6695 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:10:17.205817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.205755    6695 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:10:17.205915 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.205818    6695 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:10:17.214777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.214757    6695 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:10:17.215167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.214989    6695 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:10:17.225843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.225821    6695 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:10:17.227774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.227754    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:10:17.228215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.228186    6695 generic.go:356] "Generic (PLEG): container finished" podID="054fd709-2d73-4ce0-ab94-0df9fbd1921a" containerID="b4ee4c31686215bbe112a5c83cd19b1745ea4beb85592d194434863d6e5ee34c" exitCode=1
Oct 30 21:10:17.238532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.238511    6695 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:10:17.240672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.240634    6695 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:10:17.245291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.245260    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1" exitCode=0
Oct 30 21:10:17.260857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.260822    6695 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="9946377acb35e3fea426c2217bb977b6e1534ecc1e2934b830b1b2a83f7cf1b6" exitCode=0
Oct 30 21:10:17.260857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.260852    6695 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="9af294d17ef00892871c53310ae35b0abdb7a383931013d0545b964ef767ab2f" exitCode=0
Oct 30 21:10:17.260857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.260859    6695 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="8c52aa936acc503ca6b37772ac6d2e868d6a496d2ce2a470d64fb1e1a2c15461" exitCode=0
Oct 30 21:10:17.261020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.260867    6695 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="17b34e55c4ea1b80af62ebf7a4ac76448f02aecca4b546c9e252b68c47aeb364" exitCode=0
Oct 30 21:10:17.261020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.260873    6695 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="5945605820d182f067de37b5bbb6af5ddca58ff0aef2968c3eff2a5a37d060fc" exitCode=0
Oct 30 21:10:17.261020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.260879    6695 generic.go:356] "Generic (PLEG): container finished" podID="19448ad0-e5a8-4760-b4c5-e8410e14af1b" containerID="8be45d996c7e3f461656858749790ad2c55abc52abfac64a3e6f5dea8e772ba2" exitCode=0
Oct 30 21:10:17.264327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.264299    6695 generic.go:356] "Generic (PLEG): container finished" podID="cf70e9740ecdd5b859325e61fd6d5f34" containerID="f73ddb974843f2b8547d06fa25901644f9bc939c371398b81e91494fcb60b08d" exitCode=0
Oct 30 21:10:17.266596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.266575    6695 generic.go:356] "Generic (PLEG): container finished" podID="8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e" containerID="5d9fdca04fc0bbbc4750fd5cc543405a0f7d336ac4c65a42d3f8f5937fe68f77" exitCode=0
Oct 30 21:10:17.295854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.295784    6695 manager.go:324] Recovery completed
Oct 30 21:10:17.306248 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.306222    6695 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:10:17.319804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.319779    6695 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:10:17.319804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.319798    6695 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:10:17.319941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.319818    6695 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:10:17.319995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.319978    6695 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:10:17.320078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.319996    6695 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:10:17.320078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320018    6695 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:10:17.320078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320026    6695 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:10:17.320078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320034    6695 policy_none.go:50] "Start"
Oct 30 21:10:17.320078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320047    6695 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:10:17.320078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320060    6695 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:10:17.320285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320184    6695 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:10:17.320285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320190    6695 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:10:17.320285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.320196    6695 policy_none.go:44] "Start"
Oct 30 21:10:17.324833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.324808    6695 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:10:17.327653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.327629    6695 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:10:17.328141 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.327666    6695 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:10:17.328201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328144    6695 server.go:87] "Starting device plugin registration server"
Oct 30 21:10:17.328396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328386    6695 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:10:17.328438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328399    6695 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:10:17.328499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328479    6695 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:10:17.328613 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328576    6695 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:10:17.328664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328635    6695 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:10:17.328664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328643    6695 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:10:17.328767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.328682    6695 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:10:17.328597525Z","UUID":"2ca2290e-f87c-4ad0-87df-b349a3e2bee2","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:10:17.330065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.330048    6695 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:10:17.330149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.330070    6695 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:10:17.330585 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.330514    6695 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:10:17.429399 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.429372    6695 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:10:17.431141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.431124    6695 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:10:17.431203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.431158    6695 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:10:17.431203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.431169    6695 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:10:17.431276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.431208    6695 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.463352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.463330    6695 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.463424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.463405    6695 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.507157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.507107    6695 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"]
Oct 30 21:10:17.507833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.507817    6695 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.507910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.507897    6695 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.507998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.507896    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal" event={"ID":"d772f23118a807c4f6e4705c5121a0bf","Type":"ContainerStarted","Data":"ccdb38768ebfbab61c0269db83345e4eef09980a8f9c47eb70feb80d7e782a56"}
Oct 30 21:10:17.507998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.507960    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal" event={"ID":"d772f23118a807c4f6e4705c5121a0bf","Type":"ContainerStarted","Data":"4466932adbab3c48214ecb4527a4de797bc873950173e730182de8ed72c9710d"}
Oct 30 21:10:17.508077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.508066    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" event={"ID":"cf70e9740ecdd5b859325e61fd6d5f34","Type":"ContainerStarted","Data":"9e141d04f7f14e3e9c98ab973bc237b9916c8e5b878c8776af5e8a881ef75a88"}
Oct 30 21:10:17.508116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.508084    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" event={"ID":"cf70e9740ecdd5b859325e61fd6d5f34","Type":"ContainerDied","Data":"f73ddb974843f2b8547d06fa25901644f9bc939c371398b81e91494fcb60b08d"}
Oct 30 21:10:17.508116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.508097    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal" event={"ID":"cf70e9740ecdd5b859325e61fd6d5f34","Type":"ContainerStarted","Data":"32371109446a8c3e51cbab1f7921a6061016750c0ae47dbcecfdeb74cd0ad70b"}
Oct 30 21:10:17.532826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.532806    6695 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:17.532895 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.532857    6695 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.533228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.533215    6695 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:17.533259 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:17.533250    6695 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.617660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.617564    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d772f23118a807c4f6e4705c5121a0bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" (UID: \"d772f23118a807c4f6e4705c5121a0bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.617660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.617628    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.617807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.617675    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.718184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.718149    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.718184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.718186    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.718410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.718202    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d772f23118a807c4f6e4705c5121a0bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" (UID: \"d772f23118a807c4f6e4705c5121a0bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.718410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.718256    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d772f23118a807c4f6e4705c5121a0bf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" (UID: \"d772f23118a807c4f6e4705c5121a0bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.718410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.718274    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:17.718410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:17.718280    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf70e9740ecdd5b859325e61fd6d5f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" (UID: \"cf70e9740ecdd5b859325e61fd6d5f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:18.153937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.153902    6695 apiserver.go:52] "Watching apiserver"
Oct 30 21:10:18.204380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.204339    6695 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:10:18.205184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.205150    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-sm2w6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg","openshift-image-registry/node-ca-8v7dh","openshift-insights/insights-runtime-extractor-7tff6","openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh","openshift-dns/dns-default-zwsqg","openshift-monitoring/metrics-server-54db77678f-lbdmq","openshift-network-operator/iptables-alerter-nb5hx","openshift-multus/network-metrics-daemon-fbzn9","kube-system/konnectivity-agent-tn775","openshift-network-diagnostics/network-check-target-pb8b2","kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal","kube-system/global-pull-secret-syncer-8w46l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal","openshift-monitoring/node-exporter-8tkj4","openshift-ovn-kubernetes/ovnkube-node-x52r2","openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw","openshift-ingress-canary/ingress-canary-slnfx","openshift-multus/multus-additional-cni-plugins-k57t9","openshift-cluster-node-tuning-operator/tuned-svkg2","openshift-console/console-85d787b45d-5wm46","openshift-dns/node-resolver-9bv86","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"]
Oct 30 21:10:18.220875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.220825    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:10:18.220875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.220871    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:10:18.221140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.220891    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.221140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.220927    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.221140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.220925    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.221140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.220945    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.221140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221001    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221184    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221276    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221316    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221322    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221324    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221361    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221369    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nxx4w\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221403    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221405    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221370    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjvtw\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221427    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t284x\""
Oct 30 21:10:18.221434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221442    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:10:18.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221453    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8m2s\""
Oct 30 21:10:18.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221467    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5899\""
Oct 30 21:10:18.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221936    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:10:18.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221938    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:10:18.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221976    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\""
Oct 30 21:10:18.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.221991    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d6fgf\""
Oct 30 21:10:18.229206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.229186    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2tg5\""
Oct 30 21:10:18.229343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.229231    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:10:18.229343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.229300    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.229726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.229708    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.230360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230342    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w6j9l\""
Oct 30 21:10:18.230443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230350    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:10:18.230492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230458    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:10:18.230492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230482    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8rdf8\""
Oct 30 21:10:18.230779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230763    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:10:18.230871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230762    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jv27h\""
Oct 30 21:10:18.230871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230765    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6qvgx\""
Oct 30 21:10:18.230871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230822    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\""
Oct 30 21:10:18.230871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.230835    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:10:18.231207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231150    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:10:18.231299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231283    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.231404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231391    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hnk74\""
Oct 30 21:10:18.231464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231395    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.231464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231440    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.231535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231484    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.231983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231918    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:10:18.231983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231924    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.231983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231947    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.231983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231962    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:10:18.232198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.231966    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:10:18.232198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232006    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:10:18.232198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232086    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:10:18.232198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232119    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:10:18.232198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232142    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.232411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232389    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.232592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232579    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.232660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232589    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:10:18.232660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232581    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:10:18.232739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232722    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:10:18.232786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232727    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.232786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232782    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:10:18.232880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.232869    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:10:18.233053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.233039    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:10:18.233135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.233122    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:10:18.234283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234256    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:10:18.234393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234353    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:10:18.234393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234384    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234387    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234419    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234452    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234460    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234473    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234421    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:10:18.234506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234495    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wc7hh\""
Oct 30 21:10:18.234741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234511    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28pwt\""
Oct 30 21:10:18.234741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.234561    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2rp7h\""
Oct 30 21:10:18.240894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.240875    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:10:18.274995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.274962    6695 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:10:18.275509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.275489    6695 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:18.276070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.276051    6695 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:18.297909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.297880    6695 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:18.298077 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.297939    6695 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-232.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:18.298283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.298261    6695 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:18.298324 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.298293    6695 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-232.ec2.internal"
Oct 30 21:10:18.320257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320229    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-cnibin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.320373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320279    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-hostroot\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.320373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320306    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6n7c\" (UniqueName: \"kubernetes.io/projected/7baeebaf-f8c0-4643-b921-fff75980046a-kube-api-access-f6n7c\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:10:18.320451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320363    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-kubelet\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.320451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320392    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-netd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.320451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320415    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.320581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320449    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.320581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320529    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.320581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320567    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.320786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320599    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-crio-socket\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.320786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320651    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34a0ac7b-1822-4157-9438-d76e6783177e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:10:18.320786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320676    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cnibin\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.320786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320701    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-var-lib-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.320786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320755    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-multus-certs\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.320988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320811    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88e4b3aa-5e27-4609-ae83-e74875a7e8da-iptables-alerter-script\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.320988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320821    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.320988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320921    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.320988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320957    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-cni-binary-copy\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.320988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.320984    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-sys\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321012    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgkt\" (UniqueName: \"kubernetes.io/projected/34a0ac7b-1822-4157-9438-d76e6783177e-kube-api-access-5hgkt\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321040    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-systemd\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321064    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-kubelet\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321084    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321091    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321116    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321123    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s8jwz\""
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321063    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88e4b3aa-5e27-4609-ae83-e74875a7e8da-iptables-alerter-script\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321151    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-socket-dir-parent\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321197    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321221    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:10:18.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321243    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321255    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321273    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321280    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321284    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kb9ct\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321301    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321329    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321352    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321368    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321387    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-cni-binary-copy\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321397    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bcg3387hou02k\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321413    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321453    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8dcb97lchjofv\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321491    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-89vxb\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321492    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-multus\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321538    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-knzzg\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321577    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5f5\" (UniqueName: \"kubernetes.io/projected/900d3f13-ab1c-478e-9851-a70264d92012-kube-api-access-cs5f5\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321593    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321631    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321638    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-node-log\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321623    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321693    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-dbus\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321729    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-grpc-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321765    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgz58\" (UniqueName: \"kubernetes.io/projected/a2806653-74fd-4d37-a4ba-459232f41e74-kube-api-access-jgz58\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321790    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321865    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-data-volume\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321902    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321973    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-grpc-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321983    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.321990    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-data-volume\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322029    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.322098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322060    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb96b\" (UniqueName: \"kubernetes.io/projected/67790f00-7ce5-4f74-ae11-893cf8e217ad-kube-api-access-pb96b\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322123    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322189    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-tls\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322212    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-bin\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322231    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-config\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322256    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-env-overrides\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322257    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322271    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322307    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-metrics-server-audit-profiles\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322337    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-conf\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322367    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322408    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-socket-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322450    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-config\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322468    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322484    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-tls\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322496    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-netns\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322510    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-env-overrides\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322547    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322566    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-multus-daemon-config\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322596    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c567c895-2164-4fdc-a4b4-4df9af394efc-konnectivity-ca\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322595    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2eed9c68-fae8-40b3-9bee-499e396d96e5-metrics-server-audit-profiles\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322707    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322730    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-root\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322741    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a33b283-c453-4742-9142-173f02cadeab-multus-daemon-config\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322800    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-kubernetes\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322834    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322843    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c567c895-2164-4fdc-a4b4-4df9af394efc-konnectivity-ca\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322876    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7baeebaf-f8c0-4643-b921-fff75980046a-cert\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.322975    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ss9\" (UniqueName: \"kubernetes.io/projected/67203193-932d-445f-b14d-457040dcee8e-kube-api-access-c4ss9\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323013    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323083    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274cp\" (UniqueName: \"kubernetes.io/projected/054fd709-2d73-4ce0-ab94-0df9fbd1921a-kube-api-access-274cp\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323136    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7baeebaf-f8c0-4643-b921-fff75980046a-cert\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323173    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2eed9c68-fae8-40b3-9bee-499e396d96e5-audit-log\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323190    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d13165-1255-4e98-b0c8-a84b78043c93-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323237    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323261    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-host\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323312    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2eed9c68-fae8-40b3-9bee-499e396d96e5-audit-log\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.323358    6695 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323356    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-etc-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323380    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d13165-1255-4e98-b0c8-a84b78043c93-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.323469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323438    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.323511    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.823469619 +0000 UTC m=+1.745927911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323535    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323648    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323650    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca7df3d-5d59-436f-a381-a0224e48272f-tmp-dir\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323681    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-client-certs\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323699    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxwn\" (UniqueName: \"kubernetes.io/projected/93b394b4-d048-4139-a5eb-c32918e048d2-kube-api-access-rcxwn\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323716    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c567c895-2164-4fdc-a4b4-4df9af394efc-agent-certs\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323721    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca7df3d-5d59-436f-a381-a0224e48272f-tmp-dir\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323741    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhqr\" (UniqueName: \"kubernetes.io/projected/88e4b3aa-5e27-4609-ae83-e74875a7e8da-kube-api-access-cwhqr\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323847    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323864    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-metrics-client-ca\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323900    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-sys-fs\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323916    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-client-certs\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323922    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c567c895-2164-4fdc-a4b4-4df9af394efc-agent-certs\") pod \"konnectivity-agent-tn775\" (UID: \"c567c895-2164-4fdc-a4b4-4df9af394efc\") " pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323932    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s697g\" (UniqueName: \"kubernetes.io/projected/2eed9c68-fae8-40b3-9bee-499e396d96e5-kube-api-access-s697g\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.323983    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-k8s-cni-cncf-io\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324015    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324049    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-metrics-client-ca\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324053    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ca7df3d-5d59-436f-a381-a0224e48272f-metrics-tls\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.324086    6695 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324133    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b394b4-d048-4139-a5eb-c32918e048d2-tmp-dir\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.324151    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.824138827 +0000 UTC m=+1.746597121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"service-ca" not registered
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-kubelet-config\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324193    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysconfig\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324207    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b394b4-d048-4139-a5eb-c32918e048d2-tmp-dir\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324268    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67203193-932d-445f-b14d-457040dcee8e-serviceca\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324291    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ca7df3d-5d59-436f-a381-a0224e48272f-metrics-tls\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324324    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67790f00-7ce5-4f74-ae11-893cf8e217ad-tmp\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324341    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-tls\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324359    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-tmp\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324381    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324411    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324428    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67790f00-7ce5-4f74-ae11-893cf8e217ad-tmp\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324452    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-tmp\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324488    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67790f00-7ce5-4f74-ae11-893cf8e217ad-metrics-client-ca\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324516    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcdb\" (UniqueName: \"kubernetes.io/projected/9a33b283-c453-4742-9142-173f02cadeab-kube-api-access-4fcdb\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324543    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b394b4-d048-4139-a5eb-c32918e048d2-hosts-file\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324573    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-secret-metrics-server-tls\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324598    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324627    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324574    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324668    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324663    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-tls\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324699    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324717    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67790f00-7ce5-4f74-ae11-893cf8e217ad-metrics-client-ca\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324733    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324772    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-lib-modules\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324786    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324802    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkxm\" (UniqueName: \"kubernetes.io/projected/19448ad0-e5a8-4760-b4c5-e8410e14af1b-kube-api-access-2nkxm\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324836    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-device-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67203193-932d-445f-b14d-457040dcee8e-serviceca\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324969    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-sys\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.324988    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325003    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325048    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-system-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325083    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325096    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325110    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-wtmp\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.325177 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325216    6695 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325228    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-modprobe-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325268    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.82525475 +0000 UTC m=+1.747713030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-config" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325272    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325281    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900d3f13-ab1c-478e-9851-a70264d92012-metrics-certs\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325302    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325323    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325340    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7twc\" (UniqueName: \"kubernetes.io/projected/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-kube-api-access-j7twc\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325355    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-slash\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325324    6695 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325376    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325381    6695 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325399    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325422    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.825405521 +0000 UTC m=+1.747863822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325443    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.825436927 +0000 UTC m=+1.747895203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325461    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325483    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88e4b3aa-5e27-4609-ae83-e74875a7e8da-host-slash\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325509    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325563    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-textfile\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325592    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325628    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trphj\" (UniqueName: \"kubernetes.io/projected/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-api-access-trphj\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325640    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-textfile\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325647    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325653    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325677    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325699    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-run\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325730    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325751    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325765    6695 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325767    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-registration-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325841    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325867    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.325917    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.825905222 +0000 UTC m=+1.748363498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325945    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-bin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325969    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325965    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3d13165-1255-4e98-b0c8-a84b78043c93-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325965    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.325999    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-var-lib-kubelet\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326026    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-systemd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326045    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovn-node-metrics-cert\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326060    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-script-lib\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326124    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-netns\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326176    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxvk\" (UniqueName: \"kubernetes.io/projected/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kube-api-access-khxvk\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326212    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-log-socket\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326246    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326273    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovnkube-script-lib\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326274    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326297    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/054fd709-2d73-4ce0-ab94-0df9fbd1921a-ovn-node-metrics-cert\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326325    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-etc-kubernetes\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326347    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjxk\" (UniqueName: \"kubernetes.io/projected/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-api-access-tkjxk\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326346    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c61a330-f062-4cbf-9794-3f7973e3826d-original-pull-secret\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326364    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9br6h\" (UniqueName: \"kubernetes.io/projected/b3d13165-1255-4e98-b0c8-a84b78043c93-kube-api-access-9br6h\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326411    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326480    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-system-cni-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326508    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-ovn\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326539    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fl2\" (UniqueName: \"kubernetes.io/projected/2ca7df3d-5d59-436f-a381-a0224e48272f-kube-api-access-s9fl2\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326588    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326655    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-conf-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326689    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326651    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326709    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326659    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326716    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-etc-tuned\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326748    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326764    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2806653-74fd-4d37-a4ba-459232f41e74-etc-tuned\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326791    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326821    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67203193-932d-445f-b14d-457040dcee8e-host\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326847    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-client-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326874    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326899    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326916    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326933    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-systemd-units\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326967    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19448ad0-e5a8-4760-b4c5-e8410e14af1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.326971    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca7df3d-5d59-436f-a381-a0224e48272f-config-volume\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.327013    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.327061    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2eed9c68-fae8-40b3-9bee-499e396d96e5-client-ca-bundle\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.327067    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.327147    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca7df3d-5d59-436f-a381-a0224e48272f-config-volume\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.327553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.327189    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67790f00-7ce5-4f74-ae11-893cf8e217ad-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.331214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.331196    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.331271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.331204    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34a0ac7b-1822-4157-9438-d76e6783177e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:10:18.339117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.339093    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgkt\" (UniqueName: \"kubernetes.io/projected/34a0ac7b-1822-4157-9438-d76e6783177e-kube-api-access-5hgkt\") pod \"monitoring-plugin-77f4f76594-4x9mh\" (UID: \"34a0ac7b-1822-4157-9438-d76e6783177e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:10:18.339219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.339099    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6n7c\" (UniqueName: \"kubernetes.io/projected/7baeebaf-f8c0-4643-b921-fff75980046a-kube-api-access-f6n7c\") pod \"ingress-canary-slnfx\" (UID: \"7baeebaf-f8c0-4643-b921-fff75980046a\") " pod="openshift-ingress-canary/ingress-canary-slnfx"
Oct 30 21:10:18.339430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.339405    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgz58\" (UniqueName: \"kubernetes.io/projected/a2806653-74fd-4d37-a4ba-459232f41e74-kube-api-access-jgz58\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.339934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.339915    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb96b\" (UniqueName: \"kubernetes.io/projected/67790f00-7ce5-4f74-ae11-893cf8e217ad-kube-api-access-pb96b\") pod \"thanos-querier-85f94b8bfc-8cgkb\" (UID: \"67790f00-7ce5-4f74-ae11-893cf8e217ad\") " pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:18.340073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.340056    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5f5\" (UniqueName: \"kubernetes.io/projected/900d3f13-ab1c-478e-9851-a70264d92012-kube-api-access-cs5f5\") pod \"network-metrics-daemon-fbzn9\" (UID: \"900d3f13-ab1c-478e-9851-a70264d92012\") " pod="openshift-multus/network-metrics-daemon-fbzn9"
Oct 30 21:10:18.343334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.343311    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ss9\" (UniqueName: \"kubernetes.io/projected/67203193-932d-445f-b14d-457040dcee8e-kube-api-access-c4ss9\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.343508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.343477    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r9r\" (UniqueName: \"kubernetes.io/projected/38c2f0bf-d12e-497e-b64b-220761a147a6-kube-api-access-q4r9r\") pod \"network-check-target-pb8b2\" (UID: \"38c2f0bf-d12e-497e-b64b-220761a147a6\") " pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:18.343581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.343556    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhqr\" (UniqueName: \"kubernetes.io/projected/88e4b3aa-5e27-4609-ae83-e74875a7e8da-kube-api-access-cwhqr\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.343651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.343582    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274cp\" (UniqueName: \"kubernetes.io/projected/054fd709-2d73-4ce0-ab94-0df9fbd1921a-kube-api-access-274cp\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.345833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.345815    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s697g\" (UniqueName: \"kubernetes.io/projected/2eed9c68-fae8-40b3-9bee-499e396d96e5-kube-api-access-s697g\") pod \"metrics-server-54db77678f-lbdmq\" (UID: \"2eed9c68-fae8-40b3-9bee-499e396d96e5\") " pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:18.345926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.345876    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9br6h\" (UniqueName: \"kubernetes.io/projected/b3d13165-1255-4e98-b0c8-a84b78043c93-kube-api-access-9br6h\") pod \"openshift-state-metrics-7bd7489957-86ht2\" (UID: \"b3d13165-1255-4e98-b0c8-a84b78043c93\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-86ht2"
Oct 30 21:10:18.346016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.345999    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fl2\" (UniqueName: \"kubernetes.io/projected/2ca7df3d-5d59-436f-a381-a0224e48272f-kube-api-access-s9fl2\") pod \"dns-default-zwsqg\" (UID: \"2ca7df3d-5d59-436f-a381-a0224e48272f\") " pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:18.346093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346069    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trphj\" (UniqueName: \"kubernetes.io/projected/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-kube-api-access-trphj\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.346179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346159    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxwn\" (UniqueName: \"kubernetes.io/projected/93b394b4-d048-4139-a5eb-c32918e048d2-kube-api-access-rcxwn\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.346499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346473    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxvk\" (UniqueName: \"kubernetes.io/projected/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kube-api-access-khxvk\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.346575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346521    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkxm\" (UniqueName: \"kubernetes.io/projected/19448ad0-e5a8-4760-b4c5-e8410e14af1b-kube-api-access-2nkxm\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.346711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346689    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7twc\" (UniqueName: \"kubernetes.io/projected/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-kube-api-access-j7twc\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.346834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346814    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") pod \"alertmanager-main-0\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:18.346893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.346876    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjxk\" (UniqueName: \"kubernetes.io/projected/0d9f385c-ce7e-42d9-a79c-2ef0976f44f5-kube-api-access-tkjxk\") pod \"kube-state-metrics-5dc8d7d89b-ljnpg\" (UID: \"0d9f385c-ce7e-42d9-a79c-2ef0976f44f5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ljnpg"
Oct 30 21:10:18.347688 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.347670    6695 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:10:18.347772 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.347692    6695 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:10:18.347772 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.347707    6695 projected.go:196] Error preparing data for projected volume kube-api-access-rj6p4 for pod openshift-console/console-85d787b45d-5wm46: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:10:18.347772 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.347767    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4 podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:18.847749433 +0000 UTC m=+1.770207727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6p4" (UniqueName: "kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:10:18.348140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.348123    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcdb\" (UniqueName: \"kubernetes.io/projected/9a33b283-c453-4742-9142-173f02cadeab-kube-api-access-4fcdb\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.428341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-kubelet-config\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.428341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428285    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysconfig\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428307    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b394b4-d048-4139-a5eb-c32918e048d2-hosts-file\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.428341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428329    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428350    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428389    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysconfig\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428400    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-lib-modules\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428407    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428405    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b394b4-d048-4139-a5eb-c32918e048d2-hosts-file\") pod \"node-resolver-9bv86\" (UID: \"93b394b4-d048-4139-a5eb-c32918e048d2\") " pod="openshift-dns/node-resolver-9bv86"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428404    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428392    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-kubelet-config\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428440    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-device-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428474    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-sys\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428495    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-system-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428511    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-lib-modules\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428516    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-sys\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428474    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-device-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428538    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-wtmp\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428552    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-system-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428565    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-modprobe-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428645    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-slash\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428661    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.428668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428670    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-node-exporter-wtmp\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428677    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428701    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428712    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88e4b3aa-5e27-4609-ae83-e74875a7e8da-host-slash\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428716    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-slash\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428671    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-modprobe-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428734    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428755    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-run\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428775    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-registration-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428803    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-bin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428806    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88e4b3aa-5e27-4609-ae83-e74875a7e8da-host-slash\") pod \"iptables-alerter-nb5hx\" (UID: \"88e4b3aa-5e27-4609-ae83-e74875a7e8da\") " pod="openshift-network-operator/iptables-alerter-nb5hx"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428811    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-run\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428819    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-var-lib-kubelet\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428835    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-systemd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428836    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-bin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428854    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-registration-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428861    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-netns\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428884    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-var-lib-kubelet\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-netns\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428891    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-systemd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428906    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-log-socket\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428924    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-etc-kubernetes\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428948    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-system-cni-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428953    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-log-socket\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428964    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-ovn\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428979    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-etc-kubernetes\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429005    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-conf-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429006    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-system-cni-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.428981    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-conf-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429036    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-run-ovn\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429055    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67203193-932d-445f-b14d-457040dcee8e-host\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429077    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429091    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-systemd-units\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429109    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-cnibin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429137    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-hostroot\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429143    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429147    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-systemd-units\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429156    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67203193-932d-445f-b14d-457040dcee8e-host\") pod \"node-ca-8v7dh\" (UID: \"67203193-932d-445f-b14d-457040dcee8e\") " pod="openshift-image-registry/node-ca-8v7dh"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429163    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-kubelet\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429179    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-netd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429180    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-cnibin\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429182    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-hostroot\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429203    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-kubelet\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429233    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-netd\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429233    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429255    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429258    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429284    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-crio-socket\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429317    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c05ea68-c44a-4fbc-bda8-e6cb9ea15327-crio-socket\") pod \"insights-runtime-extractor-7tff6\" (UID: \"1c05ea68-c44a-4fbc-bda8-e6cb9ea15327\") " pod="openshift-insights/insights-runtime-extractor-7tff6"
Oct 30 21:10:18.429320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429351    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-cni-dir\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429363    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cnibin\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429383    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-var-lib-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429399    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-multus-certs\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429417    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-sys\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429440    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-systemd\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429446    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-var-lib-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429455    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-multus-certs\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-kubelet\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429471    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-sys\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429488    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-socket-dir-parent\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429499    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-systemd\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429521    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-multus\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429521    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-kubelet\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429505    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-var-lib-cni-multus\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429556    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-multus-socket-dir-parent\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429582    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-node-log\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429625    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-dbus\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429643    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-node-log\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429659    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-bin\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429684    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c61a330-f062-4cbf-9794-3f7973e3826d-dbus\") pod \"global-pull-secret-syncer-8w46l\" (UID: \"3c61a330-f062-4cbf-9794-3f7973e3826d\") " pod="kube-system/global-pull-secret-syncer-8w46l"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429689    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-conf\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429709    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-cni-bin\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429790    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-conf\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429837    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-socket-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429474    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19448ad0-e5a8-4760-b4c5-e8410e14af1b-cnibin\") pod \"multus-additional-cni-plugins-k57t9\" (UID: \"19448ad0-e5a8-4760-b4c5-e8410e14af1b\") " pod="openshift-multus/multus-additional-cni-plugins-k57t9"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.429716    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-socket-dir\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430178    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430204    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-netns\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430231    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-root\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430255    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-kubernetes\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430284    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-host-run-netns\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430299    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-host\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430314    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e-root\") pod \"node-exporter-8tkj4\" (UID: \"8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e\") " pod="openshift-monitoring/node-exporter-8tkj4"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430322    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-etc-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430333    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-kubernetes\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430345    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-etc-sysctl-d\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430357    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-sys-fs\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430369    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2806653-74fd-4d37-a4ba-459232f41e74-host\") pod \"tuned-svkg2\" (UID: \"a2806653-74fd-4d37-a4ba-459232f41e74\") " pod="openshift-cluster-node-tuning-operator/tuned-svkg2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430378    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/054fd709-2d73-4ce0-ab94-0df9fbd1921a-etc-openvswitch\") pod \"ovnkube-node-x52r2\" (UID: \"054fd709-2d73-4ce0-ab94-0df9fbd1921a\") " pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430383    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-k8s-cni-cncf-io\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430401    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cf6607a-ab80-47b6-ad1c-e33939b8dc1e-sys-fs\") pod \"aws-ebs-csi-driver-node-9w2bw\" (UID: \"8cf6607a-ab80-47b6-ad1c-e33939b8dc1e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9w2bw"
Oct 30 21:10:18.431143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.430468    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a33b283-c453-4742-9142-173f02cadeab-host-run-k8s-cni-cncf-io\") pod \"multus-sm2w6\" (UID: \"9a33b283-c453-4742-9142-173f02cadeab\") " pod="openshift-multus/multus-sm2w6"
Oct 30 21:10:18.833359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.833323    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.833383    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.833426    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.833450    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833463    6695 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833516    6695 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833527    6695 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833545    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.833525103 +0000 UTC m=+2.755983401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.833474    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833568    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.833558671 +0000 UTC m=+2.756016947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833572    6695 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:10:18.833578 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833516    6695 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:10:18.834043 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833632    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.833620953 +0000 UTC m=+2.756079247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:10:18.834043 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.833997    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.833972635 +0000 UTC m=+2.756430924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-config" not registered
Oct 30 21:10:18.834149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.834071    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.834212 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.834193    6695 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:10:18.834270 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.834259    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.834235622 +0000 UTC m=+2.756693905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:10:18.834386 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.834365    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.834347157 +0000 UTC m=+2.756805451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"service-ca" not registered
Oct 30 21:10:18.935041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:18.934997    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:18.935225 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.935168    6695 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:10:18.935225 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.935189    6695 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:10:18.935225 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.935200    6695 projected.go:196] Error preparing data for projected volume kube-api-access-rj6p4 for pod openshift-console/console-85d787b45d-5wm46: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:10:18.935461 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:18.935260    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4 podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:19.93524117 +0000 UTC m=+2.857699460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6p4" (UniqueName: "kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:10:19.178379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.178305    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:19.211044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.211013    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4x9mh"
Oct 30 21:10:19.372668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.372633    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zwsqg"
Oct 30 21:10:19.798578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.798545    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54db77678f-lbdmq"
Oct 30 21:10:19.844235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.844201    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.844255    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.844296    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.844322    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844331    6695 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.844345    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.844384    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.844412 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844402    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.844381372 +0000 UTC m=+4.766839648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844409    6695 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844421    6695 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844469    6695 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844478    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.844460947 +0000 UTC m=+4.766919230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"service-ca" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844499    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.84448892 +0000 UTC m=+4.766947196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-config" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844512    6695 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844518    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.84450926 +0000 UTC m=+4.766967536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:10:19.844753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844548    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.844533745 +0000 UTC m=+4.766992055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:10:19.845057 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844816    6695 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:10:19.845057 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.844865    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.84484685 +0000 UTC m=+4.767305125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:10:19.912723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.912694    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85f94b8bfc-8cgkb"
Oct 30 21:10:19.945808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:19.945770    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:19.946013 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.945991    6695 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:10:19.946078 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.946018    6695 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:10:19.946078 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.946033    6695 projected.go:196] Error preparing data for projected volume kube-api-access-rj6p4 for pod openshift-console/console-85d787b45d-5wm46: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:10:19.946161 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:19.946108    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4 podName:71a6179c-e8ae-4870-a41e-29e65a6373e1 nodeName:}" failed. No retries permitted until 2026-10-30 21:10:21.946086237 +0000 UTC m=+4.868544517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rj6p4" (UniqueName: "kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4") pod "console-85d787b45d-5wm46" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:10:21.039125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.039086    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7945d9969f-4ptgk"]
Oct 30 21:10:21.042398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.042373    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.059161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.059137    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:10:21.070976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.070940    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:10:21.071152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.071128    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:10:21.074491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.074467    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7945d9969f-4ptgk"]
Oct 30 21:10:21.097836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.097802    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:10:21.097984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.097865    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v2rbx\""
Oct 30 21:10:21.097984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.097912    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:10:21.097984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.097937    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:10:21.157026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.156985    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-serving-cert\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157024    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-tmp\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157160    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2g7\" (UniqueName: \"kubernetes.io/projected/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-kube-api-access-zx2g7\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157215    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-config\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157279    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-oauth-serving-cert\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157398    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-trusted-ca-bundle\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157424    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-oauth-config\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.157462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.157448    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-service-ca\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.159936 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.159922    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:10:21.160128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.160112    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:10:21.216655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.216625    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:10:21.258891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.258854    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-trusted-ca-bundle\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.258891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.258892    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-oauth-config\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.258912    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-service-ca\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.258933    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-serving-cert\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.258994    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-tmp\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259076    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2g7\" (UniqueName: \"kubernetes.io/projected/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-kube-api-access-zx2g7\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259134    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-config\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259188    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-oauth-serving-cert\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259343    6695 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:10:21.259578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259544    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-tmp\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259782    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-service-ca\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259898    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-oauth-serving-cert\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.259984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.259933    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-config\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.260178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.260156    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-trusted-ca-bundle\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.262315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.262283    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-serving-cert\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.262722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.262705    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-console-oauth-config\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.277951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.277919    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2g7\" (UniqueName: \"kubernetes.io/projected/c1cde09f-1065-44f4-896a-a2ec6c6adcb8-kube-api-access-zx2g7\") pod \"console-7945d9969f-4ptgk\" (UID: \"c1cde09f-1065-44f4-896a-a2ec6c6adcb8\") " pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.353246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.353153    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:21.651177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.651083    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7945d9969f-4ptgk"]
Oct 30 21:10:21.656545 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:21.656500    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cde09f_1065_44f4_896a_a2ec6c6adcb8.slice/crio-05a208039e2bd0596c08c8784727a080905e95cd7f8df8552e23ff15e88c713f WatchSource:0}: Error finding container 05a208039e2bd0596c08c8784727a080905e95cd7f8df8552e23ff15e88c713f: Status 404 returned error can't find the container with id 05a208039e2bd0596c08c8784727a080905e95cd7f8df8552e23ff15e88c713f
Oct 30 21:10:21.865204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865167    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865212    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865255    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865283    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865353    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865396    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865477    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865506    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865638    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865650    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.865918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.865747    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.866043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.866023    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.966522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.966436    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:21.967203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:21.967183    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"console-85d787b45d-5wm46\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") " pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:22.291734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:22.291692    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7945d9969f-4ptgk" event={"ID":"c1cde09f-1065-44f4-896a-a2ec6c6adcb8","Type":"ContainerStarted","Data":"f56094896d9e1d06a6c3911411a499ddcc76afc8e4514b69283cb1ac8189ec18"}
Oct 30 21:10:22.292105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:22.291758    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7945d9969f-4ptgk" event={"ID":"c1cde09f-1065-44f4-896a-a2ec6c6adcb8","Type":"ContainerStarted","Data":"05a208039e2bd0596c08c8784727a080905e95cd7f8df8552e23ff15e88c713f"}
Oct 30 21:10:22.330876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:22.330822    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7945d9969f-4ptgk" podStartSLOduration=1.330808206 podStartE2EDuration="1.330808206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:22.330790602 +0000 UTC m=+5.253248900" watchObservedRunningTime="2026-10-30 21:10:22.330808206 +0000 UTC m=+5.253266498"
Oct 30 21:10:23.138124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:23.138094    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tn775"
Oct 30 21:10:24.310491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:24.310457    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:24.310960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:24.310728    6695 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:10:25.357412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.357381    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:10:25.357906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.357693    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="alertmanager" containerID="cri-o://045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075" gracePeriod=120
Oct 30 21:10:25.357906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.357769    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-metric" containerID="cri-o://4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab" gracePeriod=120
Oct 30 21:10:25.357906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.357815    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="prom-label-proxy" containerID="cri-o://eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768" gracePeriod=120
Oct 30 21:10:25.357906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.357847    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy" containerID="cri-o://f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2" gracePeriod=120
Oct 30 21:10:25.358149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.358004    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-web" containerID="cri-o://b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9" gracePeriod=120
Oct 30 21:10:25.358149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.358084    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="config-reloader" containerID="cri-o://a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261" gracePeriod=120
Oct 30 21:10:25.395519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:25.395488    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:26.309870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309835    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768" exitCode=0
Oct 30 21:10:26.309870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309862    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab" exitCode=0
Oct 30 21:10:26.309870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309868    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2" exitCode=0
Oct 30 21:10:26.309870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309874    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261" exitCode=0
Oct 30 21:10:26.310129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309881    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075" exitCode=0
Oct 30 21:10:26.310129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768"}
Oct 30 21:10:26.310129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309968    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab"}
Oct 30 21:10:26.310129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.309996    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2"}
Oct 30 21:10:26.310129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.310020    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261"}
Oct 30 21:10:26.310129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.310042    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075"}
Oct 30 21:10:26.616293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.616265    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:26.712811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.712779    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.712835    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.712872    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713043    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713105    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713140    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713172    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713193    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:26.713221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713215    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713247    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:26.713413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713279    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713312    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713340    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713366    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713444    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") pod \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\" (UID: \"c06a95f5-9691-4b5d-b247-1c2c0d524f5d\") "
Oct 30 21:10:26.713677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713593    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:26.713777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713749    6695 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-main-db\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.713777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713769    6695 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-metrics-client-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.713876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.713784    6695 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.717694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.717659    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:26.717988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.717935    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.718375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.718330    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.718714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.718666    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "kube-api-access-ntt6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:26.718992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.718968    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:26.719123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.718974    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.719123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.719030    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.719558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.719522    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.723085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.723059    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.729332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.729301    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config" pod "c06a95f5-9691-4b5d-b247-1c2c0d524f5d" (UID: "c06a95f5-9691-4b5d-b247-1c2c0d524f5d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:26.814759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814710    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntt6d\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-kube-api-access-ntt6d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.814759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814749    6695 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-cluster-tls-config\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.814759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814763    6695 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-main-tls\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814776    6695 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-tls-assets\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814792    6695 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814805    6695 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-out\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814817    6695 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814829    6695 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814841    6695 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-web-config\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:26.815017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:26.814853    6695 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c06a95f5-9691-4b5d-b247-1c2c0d524f5d-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.316670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.316640    6695 generic.go:356] "Generic (PLEG): container finished" podID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerID="b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9" exitCode=0
Oct 30 21:10:27.316843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.316701    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9"}
Oct 30 21:10:27.316843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.316747    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.316843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.316767    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c06a95f5-9691-4b5d-b247-1c2c0d524f5d","Type":"ContainerDied","Data":"f87ffc7822d3ac9f86fc04a44ac50983984264f729d2cb95cbb48235a0a55991"}
Oct 30 21:10:27.316843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.316799    6695 scope.go:122] "RemoveContainer" containerID="eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768"
Oct 30 21:10:27.327957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.327924    6695 scope.go:122] "RemoveContainer" containerID="4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab"
Oct 30 21:10:27.336836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.336815    6695 scope.go:122] "RemoveContainer" containerID="f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2"
Oct 30 21:10:27.344632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.344585    6695 scope.go:122] "RemoveContainer" containerID="b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9"
Oct 30 21:10:27.351572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.351553    6695 scope.go:122] "RemoveContainer" containerID="a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261"
Oct 30 21:10:27.358801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.358783    6695 scope.go:122] "RemoveContainer" containerID="045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075"
Oct 30 21:10:27.365993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.365978    6695 scope.go:122] "RemoveContainer" containerID="bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1"
Oct 30 21:10:27.373827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.373811    6695 scope.go:122] "RemoveContainer" containerID="eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768"
Oct 30 21:10:27.374129 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.374106    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768\": container with ID starting with eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768 not found: ID does not exist" containerID="eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768"
Oct 30 21:10:27.374173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374140    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768"} err="failed to get container status \"eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768\": rpc error: code = NotFound desc = could not find container \"eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768\": container with ID starting with eb27f572f32b8f1d2529660f75ec58c3deaa2f79aad449f3aeb1290597a65768 not found: ID does not exist"
Oct 30 21:10:27.374210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374181    6695 scope.go:122] "RemoveContainer" containerID="4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab"
Oct 30 21:10:27.374426 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.374411    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab\": container with ID starting with 4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab not found: ID does not exist" containerID="4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab"
Oct 30 21:10:27.374463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374431    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab"} err="failed to get container status \"4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab\": rpc error: code = NotFound desc = could not find container \"4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab\": container with ID starting with 4a57f673b817b701c9e2eb3a5bfd2c0a14c28cba104469bf9f3110049dc184ab not found: ID does not exist"
Oct 30 21:10:27.374463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374445    6695 scope.go:122] "RemoveContainer" containerID="f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2"
Oct 30 21:10:27.374681 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.374662    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2\": container with ID starting with f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2 not found: ID does not exist" containerID="f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2"
Oct 30 21:10:27.374718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374688    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2"} err="failed to get container status \"f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2\": rpc error: code = NotFound desc = could not find container \"f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2\": container with ID starting with f19d26948c381b043d352f2a8abc451e1485cffe503568641a3f3da5a2b650d2 not found: ID does not exist"
Oct 30 21:10:27.374718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374705    6695 scope.go:122] "RemoveContainer" containerID="b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9"
Oct 30 21:10:27.374953 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.374939    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9\": container with ID starting with b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9 not found: ID does not exist" containerID="b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9"
Oct 30 21:10:27.375004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374956    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9"} err="failed to get container status \"b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9\": rpc error: code = NotFound desc = could not find container \"b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9\": container with ID starting with b618b71800b8fb80fce0f12be0538cf53be753a8a59d92d57e19c12f2fe94af9 not found: ID does not exist"
Oct 30 21:10:27.375004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.374967    6695 scope.go:122] "RemoveContainer" containerID="a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261"
Oct 30 21:10:27.375183 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.375163    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261\": container with ID starting with a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261 not found: ID does not exist" containerID="a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261"
Oct 30 21:10:27.375242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.375193    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261"} err="failed to get container status \"a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261\": rpc error: code = NotFound desc = could not find container \"a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261\": container with ID starting with a2727c90459655dbcfd768292dc90076105a321ed69c6185daea055bcd54e261 not found: ID does not exist"
Oct 30 21:10:27.375242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.375214    6695 scope.go:122] "RemoveContainer" containerID="045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075"
Oct 30 21:10:27.375454 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.375438    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075\": container with ID starting with 045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075 not found: ID does not exist" containerID="045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075"
Oct 30 21:10:27.375500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.375456    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075"} err="failed to get container status \"045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075\": rpc error: code = NotFound desc = could not find container \"045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075\": container with ID starting with 045f0cf0941a9ca4f059ddab87eda1fe97e6cdc146939dad22a6652eb9fca075 not found: ID does not exist"
Oct 30 21:10:27.375500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.375468    6695 scope.go:122] "RemoveContainer" containerID="bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1"
Oct 30 21:10:27.375683 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:27.375663    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1\": container with ID starting with bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1 not found: ID does not exist" containerID="bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1"
Oct 30 21:10:27.375728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.375690    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1"} err="failed to get container status \"bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1\": rpc error: code = NotFound desc = could not find container \"bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1\": container with ID starting with bea282e663eb95a7303ebd431346f41ad79b582a7e563e8b86ab048702a038d1 not found: ID does not exist"
Oct 30 21:10:27.382516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.382495    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:10:27.393979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.393954    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:10:27.412785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.412759    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:10:27.413159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413116    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy"
Oct 30 21:10:27.413159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413135    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy"
Oct 30 21:10:27.413159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413149    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="init-config-reloader"
Oct 30 21:10:27.413159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413155    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="init-config-reloader"
Oct 30 21:10:27.413159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413166    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="alertmanager"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413180    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="alertmanager"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413192    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-web"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413199    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-web"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413235    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-metric"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413244    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-metric"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413261    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="prom-label-proxy"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413268    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="prom-label-proxy"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413278    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="config-reloader"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413285    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="config-reloader"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413357    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-metric"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413365    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="prom-label-proxy"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413371    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="config-reloader"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413377    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413385    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="init-config-reloader"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413395    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="alertmanager"
Oct 30 21:10:27.413505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.413401    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" containerName="kube-rbac-proxy-web"
Oct 30 21:10:27.417116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.417097    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.433378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433346    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433381    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433352    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433440    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433451    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433464    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-knzzg\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433473    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433483    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:10:27.433511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.433457    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:10:27.445131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.445107    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:10:27.446227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.446206    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:10:27.521555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521510    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.521768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521648    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-config-out\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.521768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521677    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.521768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521699    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnl5h\" (UniqueName: \"kubernetes.io/projected/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-kube-api-access-dnl5h\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.521768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521732    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.521972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521861    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.521972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521901    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.522042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521967    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.522042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.521992    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.522114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.522082    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.522114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.522098    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.522180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.522118    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.522180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.522137    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-web-config\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.622996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.622949    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623004    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623030    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623052    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623074    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623131    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623157    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623189    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623217    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-web-config\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623254    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623320    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-config-out\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623345    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.623496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623368    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnl5h\" (UniqueName: \"kubernetes.io/projected/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-kube-api-access-dnl5h\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.624072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.623734    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.624462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.624425    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.624776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.624705    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.626616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.626544    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.626712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.626630    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.626712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.626668    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.627360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.627215    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.627360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.627254    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-web-config\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.627360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.627319    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.627496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.627390    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.627496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.627488    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.627755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.627736    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-config-out\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.642792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.642769    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnl5h\" (UniqueName: \"kubernetes.io/projected/2b190e5a-08c3-4cdf-b224-d96fa2023ae2-kube-api-access-dnl5h\") pod \"alertmanager-main-0\" (UID: \"2b190e5a-08c3-4cdf-b224-d96fa2023ae2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:27.733857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:27.733822    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:10:28.064275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:28.064188    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:10:28.069524 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:10:28.069496    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b190e5a_08c3_4cdf_b224_d96fa2023ae2.slice/crio-6f747031bb85515ff71571074ae7b9e46c4d484011b0fc948f43574f3f14cb20 WatchSource:0}: Error finding container 6f747031bb85515ff71571074ae7b9e46c4d484011b0fc948f43574f3f14cb20: Status 404 returned error can't find the container with id 6f747031bb85515ff71571074ae7b9e46c4d484011b0fc948f43574f3f14cb20
Oct 30 21:10:28.322879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:28.322787    6695 generic.go:356] "Generic (PLEG): container finished" podID="2b190e5a-08c3-4cdf-b224-d96fa2023ae2" containerID="0c178cd16e8328d46fc7e968408778ddc6a123187bbbc0baced77495691c41ec" exitCode=0
Oct 30 21:10:28.322879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:28.322864    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerDied","Data":"0c178cd16e8328d46fc7e968408778ddc6a123187bbbc0baced77495691c41ec"}
Oct 30 21:10:28.323079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:28.322953    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"6f747031bb85515ff71571074ae7b9e46c4d484011b0fc948f43574f3f14cb20"}
Oct 30 21:10:29.211714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.211679    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c06a95f5-9691-4b5d-b247-1c2c0d524f5d" path="/var/lib/kubelet/pods/c06a95f5-9691-4b5d-b247-1c2c0d524f5d/volumes"
Oct 30 21:10:29.331298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.331259    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"45941cf3758a5a904a106d9482ed30938e5a1bf8aaf51d5703233710c035333e"}
Oct 30 21:10:29.331494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.331328    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"b7126650ace854826ee62b7c031553a284dc8874e5f15d5719f06f572c2344ef"}
Oct 30 21:10:29.331494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.331353    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"d979632641e2a1fc665616588d08d57376769261bb0e1e6af03f4e79e39731a7"}
Oct 30 21:10:29.331494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.331378    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"2abd667a95e9121aa8ca3cfd1973de7046814175c6ba48bc015e8bc9e76e4982"}
Oct 30 21:10:29.331494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.331406    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"7950f6812bce979db37963cff2d3b5014ee3c77cea4116ad5165ad006265c94e"}
Oct 30 21:10:29.331494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.331431    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b190e5a-08c3-4cdf-b224-d96fa2023ae2","Type":"ContainerStarted","Data":"811470bcc16ac3094dbf06917b899926be9fd05318fcde376ed86108b511799c"}
Oct 30 21:10:29.387892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:29.387841    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.375694528 podStartE2EDuration="2.387825212s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.130684ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:29.38683827 +0000 UTC m=+12.309296580" watchObservedRunningTime="2026-10-30 21:10:29.387825212 +0000 UTC m=+12.310283507"
Oct 30 21:10:30.616673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:30.616638    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:30.621380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:30.621356    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:31.070988 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:10:31.070949    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:10:31.342796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:31.342717    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7945d9969f-4ptgk"
Oct 30 21:10:42.657968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:42.657936    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x52r2"
Oct 30 21:10:43.240970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.240928    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85d787b45d-5wm46" podUID="71a6179c-e8ae-4870-a41e-29e65a6373e1" containerName="console" containerID="cri-o://6548df44376feac9377eadb80dee7c214556a7ed6ed983c5c6c3820ff153360d" gracePeriod=15
Oct 30 21:10:43.387439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.387399    6695 generic.go:356] "Generic (PLEG): container finished" podID="71a6179c-e8ae-4870-a41e-29e65a6373e1" containerID="6548df44376feac9377eadb80dee7c214556a7ed6ed983c5c6c3820ff153360d" exitCode=0
Oct 30 21:10:43.387439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.387427    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d787b45d-5wm46" event={"ID":"71a6179c-e8ae-4870-a41e-29e65a6373e1","Type":"ContainerDied","Data":"6548df44376feac9377eadb80dee7c214556a7ed6ed983c5c6c3820ff153360d"}
Oct 30 21:10:43.479673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.479647    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:43.548330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548293    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548332    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548391    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548418    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548460    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548527    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548566    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548634    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") pod \"71a6179c-e8ae-4870-a41e-29e65a6373e1\" (UID: \"71a6179c-e8ae-4870-a41e-29e65a6373e1\") "
Oct 30 21:10:43.548967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548892    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:43.549061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548972    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:43.549129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548978    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:43.549129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.548966    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:43.549227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.549163    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:43.550960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.550933    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:43.550960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.550951    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "kube-api-access-rj6p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:43.551111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.550971    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config" pod "71a6179c-e8ae-4870-a41e-29e65a6373e1" (UID: "71a6179c-e8ae-4870-a41e-29e65a6373e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:43.649482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649428    6695 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-oauth-config\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649476    6695 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-oauth-serving-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649490    6695 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71a6179c-e8ae-4870-a41e-29e65a6373e1-tmp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649501    6695 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-serving-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649511    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj6p4\" (UniqueName: \"kubernetes.io/projected/71a6179c-e8ae-4870-a41e-29e65a6373e1-kube-api-access-rj6p4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649521    6695 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-console-config\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649531    6695 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-trusted-ca-bundle\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:43.649797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:43.649540    6695 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6179c-e8ae-4870-a41e-29e65a6373e1-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:10:44.393298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:44.393263    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d787b45d-5wm46"
Oct 30 21:10:44.393298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:44.393271    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d787b45d-5wm46" event={"ID":"71a6179c-e8ae-4870-a41e-29e65a6373e1","Type":"ContainerDied","Data":"8bc6e9824a7374ff1dffb29d91bb2541f11448fbd0b36e3a061d756a566ab007"}
Oct 30 21:10:44.393801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:44.393340    6695 scope.go:122] "RemoveContainer" containerID="6548df44376feac9377eadb80dee7c214556a7ed6ed983c5c6c3820ff153360d"
Oct 30 21:10:44.440028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:44.439993    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d787b45d-5wm46"]
Oct 30 21:10:44.451529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:44.451495    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85d787b45d-5wm46"]
Oct 30 21:10:45.213795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:45.213756    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a6179c-e8ae-4870-a41e-29e65a6373e1" path="/var/lib/kubelet/pods/71a6179c-e8ae-4870-a41e-29e65a6373e1/volumes"
Oct 30 21:10:51.219704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:10:51.219675    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pb8b2"
Oct 30 21:11:01.073054 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:11:01.073010    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:11:31.072818 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:11:31.072778    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:12:01.074151 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:12:01.074062    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:12:31.072342 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:12:31.072299    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:13:01.071014 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:13:01.070981    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:13:31.072535 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:13:31.072454    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:14:01.073977 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:14:01.073935    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:14:31.072033 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:14:31.071990    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:14:46.739539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.739500    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"]
Oct 30 21:14:46.740108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.739831    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71a6179c-e8ae-4870-a41e-29e65a6373e1" containerName="console"
Oct 30 21:14:46.740108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.739842    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a6179c-e8ae-4870-a41e-29e65a6373e1" containerName="console"
Oct 30 21:14:46.740108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.739898    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a6179c-e8ae-4870-a41e-29e65a6373e1" containerName="console"
Oct 30 21:14:46.743218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.743195    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.769096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.769055    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:14:46.769096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.769065    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:14:46.769286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.769139    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-q44lp\""
Oct 30 21:14:46.771596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.771574    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"]
Oct 30 21:14:46.873139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.873095    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.873139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.873138    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmdh\" (UniqueName: \"kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.873370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.873284    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.974260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.974217    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmdh\" (UniqueName: \"kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.974458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.974317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.974458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.974344    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.974822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.974795    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:46.974909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:46.974808    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:47.002171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.002099    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmdh\" (UniqueName: \"kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:47.059351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.059312    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:14:47.063755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.063734    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"]
Oct 30 21:14:47.067027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.067003    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.083189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083155    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:14:47.083346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083261    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:14:47.083346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083284    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:14:47.083915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083894    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:14:47.083915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083909    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:14:47.084051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083918    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:14:47.084051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.083942    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l5v2k\""
Oct 30 21:14:47.094965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.094930    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"]
Oct 30 21:14:47.176928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.176885    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz442\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.177101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.176941    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.177101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.176963    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.177101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.177068    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.177201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.177130    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.177201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.177183    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.177266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.177203    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278067    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278102    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278124    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278217    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz442\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278348    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278392    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.278845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.278816    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.281059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.281035    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.281165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.281084    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.281213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.281202    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.281247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.281207    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.297565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.297539    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.297744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.297726    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz442\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442\") pod \"istiod-openshift-gateway-7876bfcb74-895pj\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.371775 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:14:47.371745    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504103e3_ebcf_4b70_ba76_3a20296faf57.slice/crio-ca3c292ec5db5060b1a3ada33de089251612b00b6a4a41c963b15d8d547ef424 WatchSource:0}: Error finding container ca3c292ec5db5060b1a3ada33de089251612b00b6a4a41c963b15d8d547ef424: Status 404 returned error can't find the container with id ca3c292ec5db5060b1a3ada33de089251612b00b6a4a41c963b15d8d547ef424
Oct 30 21:14:47.373761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.373743    6695 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:14:47.387952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.387926    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"]
Oct 30 21:14:47.391888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.391866    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:47.704904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:47.704822    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"]
Oct 30 21:14:47.710354 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:14:47.710329    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda102033c_8bc4_49b9_9bbc_5c0f28f74a9b.slice/crio-4233de7f5fab543f8541b1d5f7e08003a430fb4d327872a3d5fbffba1c099483 WatchSource:0}: Error finding container 4233de7f5fab543f8541b1d5f7e08003a430fb4d327872a3d5fbffba1c099483: Status 404 returned error can't find the container with id 4233de7f5fab543f8541b1d5f7e08003a430fb4d327872a3d5fbffba1c099483
Oct 30 21:14:48.253249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:48.253189    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" event={"ID":"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b","Type":"ContainerStarted","Data":"4233de7f5fab543f8541b1d5f7e08003a430fb4d327872a3d5fbffba1c099483"}
Oct 30 21:14:48.254509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:48.254478    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn" event={"ID":"504103e3-ebcf-4b70-ba76-3a20296faf57","Type":"ContainerStarted","Data":"ca3c292ec5db5060b1a3ada33de089251612b00b6a4a41c963b15d8d547ef424"}
Oct 30 21:14:52.106525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:52.106487    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:14:52.106966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:52.106570    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:14:52.273413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:52.273371    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" event={"ID":"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b","Type":"ContainerStarted","Data":"f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6"}
Oct 30 21:14:52.274333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:52.274304    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:52.331366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:52.331297    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" podStartSLOduration=0.938195683 podStartE2EDuration="5.331278243s" totalImagesPullingTime="4.39308256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:52.330773366 +0000 UTC m=+275.253231665" watchObservedRunningTime="2026-10-30 21:14:52.331278243 +0000 UTC m=+275.253736541"
Oct 30 21:14:53.278928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:53.278892    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn" event={"ID":"504103e3-ebcf-4b70-ba76-3a20296faf57","Type":"ContainerStarted","Data":"2246f6ada884a817d3901c3c2f130e2aa064975db3eb4a44fe0ed860c9ff26d4"}
Oct 30 21:14:53.280846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:53.280811    6695 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-895pj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:14:53.280956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:53.280871    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" podUID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:14:54.284631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:54.284583    6695 generic.go:356] "Generic (PLEG): container finished" podID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerID="2246f6ada884a817d3901c3c2f130e2aa064975db3eb4a44fe0ed860c9ff26d4" exitCode=0
Oct 30 21:14:54.285108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:54.284676    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn" event={"ID":"504103e3-ebcf-4b70-ba76-3a20296faf57","Type":"ContainerDied","Data":"2246f6ada884a817d3901c3c2f130e2aa064975db3eb4a44fe0ed860c9ff26d4"}
Oct 30 21:14:54.285815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:54.285662    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:14:57.302867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:57.302830    6695 generic.go:356] "Generic (PLEG): container finished" podID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerID="8e4e287163b97bb583438039d3642651b5e2916b5b5c824f6cdaae14bb030deb" exitCode=0
Oct 30 21:14:57.303371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:14:57.302876    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn" event={"ID":"504103e3-ebcf-4b70-ba76-3a20296faf57","Type":"ContainerDied","Data":"8e4e287163b97bb583438039d3642651b5e2916b5b5c824f6cdaae14bb030deb"}
Oct 30 21:15:01.076940 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:15:01.076835    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:15:03.330273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:03.330234    6695 generic.go:356] "Generic (PLEG): container finished" podID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerID="5f4c694bcf89b162d45af2b04ee7445646ce404141b233f7b152356b76df4a22" exitCode=0
Oct 30 21:15:03.330791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:03.330321    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn" event={"ID":"504103e3-ebcf-4b70-ba76-3a20296faf57","Type":"ContainerDied","Data":"5f4c694bcf89b162d45af2b04ee7445646ce404141b233f7b152356b76df4a22"}
Oct 30 21:15:04.459406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.459381    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:15:04.638264 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.638162    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util\") pod \"504103e3-ebcf-4b70-ba76-3a20296faf57\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") "
Oct 30 21:15:04.638454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.638335    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle\") pod \"504103e3-ebcf-4b70-ba76-3a20296faf57\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") "
Oct 30 21:15:04.638454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.638370    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh\" (UniqueName: \"kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh\") pod \"504103e3-ebcf-4b70-ba76-3a20296faf57\" (UID: \"504103e3-ebcf-4b70-ba76-3a20296faf57\") "
Oct 30 21:15:04.638988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.638962    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle" pod "504103e3-ebcf-4b70-ba76-3a20296faf57" (UID: "504103e3-ebcf-4b70-ba76-3a20296faf57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:04.640610 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.640577    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh" pod "504103e3-ebcf-4b70-ba76-3a20296faf57" (UID: "504103e3-ebcf-4b70-ba76-3a20296faf57"). InnerVolumeSpecName "kube-api-access-6xmdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:04.642269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.642235    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util" pod "504103e3-ebcf-4b70-ba76-3a20296faf57" (UID: "504103e3-ebcf-4b70-ba76-3a20296faf57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:04.739777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.739735    6695 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-bundle\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:15:04.739777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.739769    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmdh\" (UniqueName: \"kubernetes.io/projected/504103e3-ebcf-4b70-ba76-3a20296faf57-kube-api-access-6xmdh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:15:04.739777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:04.739779    6695 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/504103e3-ebcf-4b70-ba76-3a20296faf57-util\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:15:05.340287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:05.340253    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn" event={"ID":"504103e3-ebcf-4b70-ba76-3a20296faf57","Type":"ContainerDied","Data":"ca3c292ec5db5060b1a3ada33de089251612b00b6a4a41c963b15d8d547ef424"}
Oct 30 21:15:05.340449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:05.340307    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3c292ec5db5060b1a3ada33de089251612b00b6a4a41c963b15d8d547ef424"
Oct 30 21:15:05.340449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:05.340271    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxqtn"
Oct 30 21:15:17.231912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:17.231885    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:15:17.236386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:17.236365    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:15:31.073882 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:15:31.073832    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:15:32.571346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571300    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"]
Oct 30 21:15:32.571887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571797    6695 cpu_manager.go:414] "containerMap: removing container" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="pull"
Oct 30 21:15:32.571887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571817    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="pull"
Oct 30 21:15:32.571887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571841    6695 cpu_manager.go:414] "containerMap: removing container" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="util"
Oct 30 21:15:32.571887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571848    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="util"
Oct 30 21:15:32.571887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571868    6695 cpu_manager.go:414] "containerMap: removing container" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="extract"
Oct 30 21:15:32.571887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571874    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="extract"
Oct 30 21:15:32.572181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.571956    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="504103e3-ebcf-4b70-ba76-3a20296faf57" containerName="extract"
Oct 30 21:15:32.574071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.574051    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.596448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.596414    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"]
Oct 30 21:15:32.657335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.657296    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.657521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.657346    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.657521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.657427    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.657521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.657482    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt76k\" (UniqueName: \"kubernetes.io/projected/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-kube-api-access-vt76k\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.657671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.657563    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.657671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.657627    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.661900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.661879    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:15:32.662349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.662333    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:15:32.662802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.662783    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:15:32.662854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.662812    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:15:32.662887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.662861    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:15:32.662935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.662920    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dggpd\""
Oct 30 21:15:32.758286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758247    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt76k\" (UniqueName: \"kubernetes.io/projected/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-kube-api-access-vt76k\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.758434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758300    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.758434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758335    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.758434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758354    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.758434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758391    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.758434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758419    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.759016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.758990    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.760918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.760884    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.761027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.760969    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.761027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.760987    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.761027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.760975    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.777844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.777817    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt76k\" (UniqueName: \"kubernetes.io/projected/01ad7552-8bcd-4650-9fb0-c9d1ffecbe27-kube-api-access-vt76k\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-w2djl\" (UID: \"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:32.890095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:32.889996    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:33.253553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:15:33.253506    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ad7552_8bcd_4650_9fb0_c9d1ffecbe27.slice/crio-4e997d7eabf324b0a18adfdf3633ad621ef1494b31a7abebc3f1ea0aaafd7ea6 WatchSource:0}: Error finding container 4e997d7eabf324b0a18adfdf3633ad621ef1494b31a7abebc3f1ea0aaafd7ea6: Status 404 returned error can't find the container with id 4e997d7eabf324b0a18adfdf3633ad621ef1494b31a7abebc3f1ea0aaafd7ea6
Oct 30 21:15:33.259550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:33.259527    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"]
Oct 30 21:15:33.441808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:33.441773    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl" event={"ID":"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27","Type":"ContainerStarted","Data":"4e997d7eabf324b0a18adfdf3633ad621ef1494b31a7abebc3f1ea0aaafd7ea6"}
Oct 30 21:15:36.457181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:36.457146    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl" event={"ID":"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27","Type":"ContainerStarted","Data":"6b777cd89164bd8c63bd962b97ea1571da7bc32c11043d63645fe6bb0d36ea23"}
Oct 30 21:15:37.463335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:37.463282    6695 generic.go:356] "Generic (PLEG): container finished" podID="01ad7552-8bcd-4650-9fb0-c9d1ffecbe27" containerID="6b777cd89164bd8c63bd962b97ea1571da7bc32c11043d63645fe6bb0d36ea23" exitCode=0
Oct 30 21:15:37.463793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:37.463380    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl" event={"ID":"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27","Type":"ContainerDied","Data":"6b777cd89164bd8c63bd962b97ea1571da7bc32c11043d63645fe6bb0d36ea23"}
Oct 30 21:15:38.469086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:38.469045    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl" event={"ID":"01ad7552-8bcd-4650-9fb0-c9d1ffecbe27","Type":"ContainerStarted","Data":"85db3be04c0f1d8063b29da7c32e7e90ebe358f3d3cabfa4ba318cbed8cbafb5"}
Oct 30 21:15:38.469519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:38.469151    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:38.523662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:38.523610    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl" podStartSLOduration=2.908781886 podStartE2EDuration="6.523569153s" totalImagesPullingTime="3.066773315s" totalInitContainerRuntime="548.013952ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:38.522420243 +0000 UTC m=+321.444878541" watchObservedRunningTime="2026-10-30 21:15:38.523569153 +0000 UTC m=+321.446027451"
Oct 30 21:15:49.475622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:49.475568    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-w2djl"
Oct 30 21:15:59.454365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.454327    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"]
Oct 30 21:15:59.458617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.458580    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.473173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.473146    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:15:59.484835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.484809    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pw6dq\""
Oct 30 21:15:59.485842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.485814    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"]
Oct 30 21:15:59.545278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.545246    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:15:59.617576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.617538    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f315049a-d615-414b-a693-5401ebb4a8f7-tmp\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.617576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.617580    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315049a-d615-414b-a693-5401ebb4a8f7-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.617860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.617618    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2lf\" (UniqueName: \"kubernetes.io/projected/f315049a-d615-414b-a693-5401ebb4a8f7-kube-api-access-rp2lf\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.718297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.718190    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f315049a-d615-414b-a693-5401ebb4a8f7-tmp\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.718297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.718245    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315049a-d615-414b-a693-5401ebb4a8f7-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.718297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.718273    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2lf\" (UniqueName: \"kubernetes.io/projected/f315049a-d615-414b-a693-5401ebb4a8f7-kube-api-access-rp2lf\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.720623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.720579    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f315049a-d615-414b-a693-5401ebb4a8f7-tmp\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.720758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.720735    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f315049a-d615-414b-a693-5401ebb4a8f7-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.749497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.749455    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2lf\" (UniqueName: \"kubernetes.io/projected/f315049a-d615-414b-a693-5401ebb4a8f7-kube-api-access-rp2lf\") pod \"kube-auth-proxy-6fccbd9b79-pg25b\" (UID: \"f315049a-d615-414b-a693-5401ebb4a8f7\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:15:59.769187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:15:59.769151    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"
Oct 30 21:16:00.070702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:00.070661    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b"]
Oct 30 21:16:00.076024 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:16:00.075987    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf315049a_d615_414b_a693_5401ebb4a8f7.slice/crio-6c9fd2f77c804817e52da4a76d8501120df77ea40f6c8bbbee4dde1fc0dc98f0 WatchSource:0}: Error finding container 6c9fd2f77c804817e52da4a76d8501120df77ea40f6c8bbbee4dde1fc0dc98f0: Status 404 returned error can't find the container with id 6c9fd2f77c804817e52da4a76d8501120df77ea40f6c8bbbee4dde1fc0dc98f0
Oct 30 21:16:00.555750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:00.555710    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b" event={"ID":"f315049a-d615-414b-a693-5401ebb4a8f7","Type":"ContainerStarted","Data":"6c9fd2f77c804817e52da4a76d8501120df77ea40f6c8bbbee4dde1fc0dc98f0"}
Oct 30 21:16:01.084823 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:16:01.084721    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:16:02.566094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:02.566054    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b" event={"ID":"f315049a-d615-414b-a693-5401ebb4a8f7","Type":"ContainerStarted","Data":"a9a6c73e42c1f47e90f5599244926321009418af67f680204489ef8e4ea0c98e"}
Oct 30 21:16:02.614213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:02.614157    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-pg25b" podStartSLOduration=1.208875931 podStartE2EDuration="3.614140954s" totalImagesPullingTime="2.405265023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:02.613977297 +0000 UTC m=+345.536435596" watchObservedRunningTime="2026-10-30 21:16:02.614140954 +0000 UTC m=+345.536599251"
Oct 30 21:16:07.364343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.364295    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"]
Oct 30 21:16:07.367080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.367056    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.381394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.381368    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-m577s\""
Oct 30 21:16:07.381594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.381529    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:16:07.383954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.383935    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3109498-853f-4884-a87c-5103affc6b25-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.384029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.383967    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3109498-853f-4884-a87c-5103affc6b25-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.384029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.383987    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3109498-853f-4884-a87c-5103affc6b25-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.384107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.384038    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjpl\" (UniqueName: \"kubernetes.io/projected/f3109498-853f-4884-a87c-5103affc6b25-kube-api-access-7sjpl\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.393099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.393072    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:16:07.417142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.417104    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 21:16:07.418105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.418078    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"]
Oct 30 21:16:07.484707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.484657    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3109498-853f-4884-a87c-5103affc6b25-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.484914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.484717    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3109498-853f-4884-a87c-5103affc6b25-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.484914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.484742    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3109498-853f-4884-a87c-5103affc6b25-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.484914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.484766    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjpl\" (UniqueName: \"kubernetes.io/projected/f3109498-853f-4884-a87c-5103affc6b25-kube-api-access-7sjpl\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.485109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.485088    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3109498-853f-4884-a87c-5103affc6b25-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.485169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.485124    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3109498-853f-4884-a87c-5103affc6b25-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.485576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.485555    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3109498-853f-4884-a87c-5103affc6b25-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.504647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.504593    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjpl\" (UniqueName: \"kubernetes.io/projected/f3109498-853f-4884-a87c-5103affc6b25-kube-api-access-7sjpl\") pod \"kserve-module-controller-manager-7566cf5fc8-zgm89\" (UID: \"f3109498-853f-4884-a87c-5103affc6b25\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.679083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.678984    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:07.990239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:07.990140    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"]
Oct 30 21:16:07.995986 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:16:07.995950    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3109498_853f_4884_a87c_5103affc6b25.slice/crio-8c787ce6b9782792407395d0f8d2bb8f44274c35f7f79ae6cd2f30f71fcecad0 WatchSource:0}: Error finding container 8c787ce6b9782792407395d0f8d2bb8f44274c35f7f79ae6cd2f30f71fcecad0: Status 404 returned error can't find the container with id 8c787ce6b9782792407395d0f8d2bb8f44274c35f7f79ae6cd2f30f71fcecad0
Oct 30 21:16:08.594213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:08.594159    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89" event={"ID":"f3109498-853f-4884-a87c-5103affc6b25","Type":"ContainerStarted","Data":"8c787ce6b9782792407395d0f8d2bb8f44274c35f7f79ae6cd2f30f71fcecad0"}
Oct 30 21:16:10.604988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:10.604948    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89" event={"ID":"f3109498-853f-4884-a87c-5103affc6b25","Type":"ContainerStarted","Data":"d72622ccb99ab0ccd75b868dcc2647770dd6a7a36ea4b0d1a97d12f4aab1d788"}
Oct 30 21:16:10.605490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:10.605057    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:10.656667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:10.656594    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89" podStartSLOduration=1.883178018 podStartE2EDuration="3.656578278s" totalImagesPullingTime="1.77340026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:10.655656145 +0000 UTC m=+353.578114444" watchObservedRunningTime="2026-10-30 21:16:10.656578278 +0000 UTC m=+353.579036598"
Oct 30 21:16:31.072870 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:16:31.072788    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:16:31.610086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:31.610050    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zgm89"
Oct 30 21:16:38.908452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:38.908415    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6ldcq"]
Oct 30 21:16:38.913080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:38.913056    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:38.929542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:38.929501    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6ldcq"]
Oct 30 21:16:38.931696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:38.931675    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8hqq2\""
Oct 30 21:16:38.940205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:38.940181    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:16:38.940205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:38.940183    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:16:39.075915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.075873    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtckd\" (UniqueName: \"kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd\") pod \"authorino-57f7f8659f-6ldcq\" (UID: \"724d1b13-52f2-4629-872f-88f5aa1c5a1d\") " pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:39.176980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.176875    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtckd\" (UniqueName: \"kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd\") pod \"authorino-57f7f8659f-6ldcq\" (UID: \"724d1b13-52f2-4629-872f-88f5aa1c5a1d\") " pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:39.196334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.196297    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtckd\" (UniqueName: \"kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd\") pod \"authorino-57f7f8659f-6ldcq\" (UID: \"724d1b13-52f2-4629-872f-88f5aa1c5a1d\") " pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:39.235444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.235404    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:39.529529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.529481    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6ldcq"]
Oct 30 21:16:39.535053 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:16:39.535020    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod724d1b13_52f2_4629_872f_88f5aa1c5a1d.slice/crio-212bfc77cb50b0bcad4c37fb17f015dcde020cf663b7475408d6e2a34a43a8b8 WatchSource:0}: Error finding container 212bfc77cb50b0bcad4c37fb17f015dcde020cf663b7475408d6e2a34a43a8b8: Status 404 returned error can't find the container with id 212bfc77cb50b0bcad4c37fb17f015dcde020cf663b7475408d6e2a34a43a8b8
Oct 30 21:16:39.685495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.685458    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r9qb2"]
Oct 30 21:16:39.688140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.688115    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:16:39.704336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.704305    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r9qb2"]
Oct 30 21:16:39.727599 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.727554    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6ldcq" event={"ID":"724d1b13-52f2-4629-872f-88f5aa1c5a1d","Type":"ContainerStarted","Data":"212bfc77cb50b0bcad4c37fb17f015dcde020cf663b7475408d6e2a34a43a8b8"}
Oct 30 21:16:39.782924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.782823    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jpb\" (UniqueName: \"kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb\") pod \"authorino-56c6c9ddd5-r9qb2\" (UID: \"4e6040aa-b37a-4715-b9f6-74cffeef487c\") " pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:16:39.884131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.884086    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jpb\" (UniqueName: \"kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb\") pod \"authorino-56c6c9ddd5-r9qb2\" (UID: \"4e6040aa-b37a-4715-b9f6-74cffeef487c\") " pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:16:39.902651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.902599    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jpb\" (UniqueName: \"kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb\") pod \"authorino-56c6c9ddd5-r9qb2\" (UID: \"4e6040aa-b37a-4715-b9f6-74cffeef487c\") " pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:16:39.999463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:39.999426    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:16:40.301951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:40.301867    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r9qb2"]
Oct 30 21:16:40.308765 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:16:40.308686    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e6040aa_b37a_4715_b9f6_74cffeef487c.slice/crio-774637cb7139162b73ba73791fdd41666a78401860cfaf8585b35441da171a68 WatchSource:0}: Error finding container 774637cb7139162b73ba73791fdd41666a78401860cfaf8585b35441da171a68: Status 404 returned error can't find the container with id 774637cb7139162b73ba73791fdd41666a78401860cfaf8585b35441da171a68
Oct 30 21:16:40.734537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:40.734456    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2" event={"ID":"4e6040aa-b37a-4715-b9f6-74cffeef487c","Type":"ContainerStarted","Data":"774637cb7139162b73ba73791fdd41666a78401860cfaf8585b35441da171a68"}
Oct 30 21:16:42.745854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:42.745816    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2" event={"ID":"4e6040aa-b37a-4715-b9f6-74cffeef487c","Type":"ContainerStarted","Data":"d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e"}
Oct 30 21:16:42.747353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:42.747324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6ldcq" event={"ID":"724d1b13-52f2-4629-872f-88f5aa1c5a1d","Type":"ContainerStarted","Data":"7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890"}
Oct 30 21:16:42.822502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:42.822381    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2" podStartSLOduration=1.560215194 podStartE2EDuration="3.822358464s" totalImagesPullingTime="2.26214327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:42.784786533 +0000 UTC m=+385.707244813" watchObservedRunningTime="2026-10-30 21:16:42.822358464 +0000 UTC m=+385.744816763"
Oct 30 21:16:42.822885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:42.822861    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6ldcq"]
Oct 30 21:16:42.837649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:42.836655    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-6ldcq" podStartSLOduration=1.790812335 podStartE2EDuration="4.836634724s" totalImagesPullingTime="3.045822389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:42.836275274 +0000 UTC m=+385.758733572" watchObservedRunningTime="2026-10-30 21:16:42.836634724 +0000 UTC m=+385.759093023"
Oct 30 21:16:44.756040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:44.755988    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-6ldcq" podUID="724d1b13-52f2-4629-872f-88f5aa1c5a1d" containerName="authorino" containerID="cri-o://7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890" gracePeriod=30
Oct 30 21:16:45.004051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.004024    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:45.134970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.134935    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd\" (UniqueName: \"kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd\") pod \"724d1b13-52f2-4629-872f-88f5aa1c5a1d\" (UID: \"724d1b13-52f2-4629-872f-88f5aa1c5a1d\") "
Oct 30 21:16:45.137240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.137209    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd" pod "724d1b13-52f2-4629-872f-88f5aa1c5a1d" (UID: "724d1b13-52f2-4629-872f-88f5aa1c5a1d"). InnerVolumeSpecName "kube-api-access-wtckd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:45.236534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.236496    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtckd\" (UniqueName: \"kubernetes.io/projected/724d1b13-52f2-4629-872f-88f5aa1c5a1d-kube-api-access-wtckd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:16:45.761707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.761672    6695 generic.go:356] "Generic (PLEG): container finished" podID="724d1b13-52f2-4629-872f-88f5aa1c5a1d" containerID="7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890" exitCode=0
Oct 30 21:16:45.762160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.761725    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-6ldcq"
Oct 30 21:16:45.762160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.761756    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6ldcq" event={"ID":"724d1b13-52f2-4629-872f-88f5aa1c5a1d","Type":"ContainerDied","Data":"7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890"}
Oct 30 21:16:45.762160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.761803    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-6ldcq" event={"ID":"724d1b13-52f2-4629-872f-88f5aa1c5a1d","Type":"ContainerDied","Data":"212bfc77cb50b0bcad4c37fb17f015dcde020cf663b7475408d6e2a34a43a8b8"}
Oct 30 21:16:45.762160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.761821    6695 scope.go:122] "RemoveContainer" containerID="7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890"
Oct 30 21:16:45.770201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.770183    6695 scope.go:122] "RemoveContainer" containerID="7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890"
Oct 30 21:16:45.770471 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:16:45.770450    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890\": container with ID starting with 7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890 not found: ID does not exist" containerID="7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890"
Oct 30 21:16:45.770515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.770482    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890"} err="failed to get container status \"7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890\": rpc error: code = NotFound desc = could not find container \"7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890\": container with ID starting with 7b2428af6d2aa3d4a2ee84d89407ecfb08f9c823644627438389487dd7dda890 not found: ID does not exist"
Oct 30 21:16:45.809200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.809166    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6ldcq"]
Oct 30 21:16:45.821345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:45.821309    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-6ldcq"]
Oct 30 21:16:47.211050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:16:47.211011    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724d1b13-52f2-4629-872f-88f5aa1c5a1d" path="/var/lib/kubelet/pods/724d1b13-52f2-4629-872f-88f5aa1c5a1d/volumes"
Oct 30 21:17:01.076922 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:17:01.076884    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:17:02.474704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.474665    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"]
Oct 30 21:17:02.475113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.475016    6695 cpu_manager.go:414] "containerMap: removing container" podUID="724d1b13-52f2-4629-872f-88f5aa1c5a1d" containerName="authorino"
Oct 30 21:17:02.475113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.475027    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724d1b13-52f2-4629-872f-88f5aa1c5a1d" containerName="authorino"
Oct 30 21:17:02.475113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.475084    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="724d1b13-52f2-4629-872f-88f5aa1c5a1d" containerName="authorino"
Oct 30 21:17:02.478385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.478355    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.503978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.503940    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"]
Oct 30 21:17:02.581800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.581757    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.581800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.581806    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdfb\" (UniqueName: \"kubernetes.io/projected/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-kube-api-access-drdfb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.582041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.581827    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.582041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.581953    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.582110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.582046    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.582110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.582065    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.582110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.582089    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.682748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.682710    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdfb\" (UniqueName: \"kubernetes.io/projected/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-kube-api-access-drdfb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.682748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.682750    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.682972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.682944    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.683070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.683052    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.683117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.683085    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.683158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.683120    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.683158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.683150    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.683715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.683686    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.685430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.685403    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.685554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.685414    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.685554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.685463    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.685554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.685528    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.701850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.701812    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.702136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.702114    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdfb\" (UniqueName: \"kubernetes.io/projected/4c5059d5-a52a-4288-abb9-a05c96f5f2dc-kube-api-access-drdfb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d2jcl\" (UID: \"4c5059d5-a52a-4288-abb9-a05c96f5f2dc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:02.790276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:02.790238    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:03.110020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.109931    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"]
Oct 30 21:17:03.115722 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:17:03.115694    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5059d5_a52a_4288_abb9_a05c96f5f2dc.slice/crio-20811f9a96f05fab791109ded7b121c98f39a7c7ee56e6ec9f576396f61d2417 WatchSource:0}: Error finding container 20811f9a96f05fab791109ded7b121c98f39a7c7ee56e6ec9f576396f61d2417: Status 404 returned error can't find the container with id 20811f9a96f05fab791109ded7b121c98f39a7c7ee56e6ec9f576396f61d2417
Oct 30 21:17:03.117944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.117911    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:17:03.118041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.117977    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:17:03.844816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.844775    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl" event={"ID":"4c5059d5-a52a-4288-abb9-a05c96f5f2dc","Type":"ContainerStarted","Data":"185f6c31cf8373ba745fa0a33bf90d6e25eb64b3ff34a2ed64fbbf286a075243"}
Oct 30 21:17:03.845200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.844833    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl" event={"ID":"4c5059d5-a52a-4288-abb9-a05c96f5f2dc","Type":"ContainerStarted","Data":"20811f9a96f05fab791109ded7b121c98f39a7c7ee56e6ec9f576396f61d2417"}
Oct 30 21:17:03.845200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.844918    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:03.890998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:03.890921    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl" podStartSLOduration=1.890900456 podStartE2EDuration="1.890900456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:03.88953746 +0000 UTC m=+406.811995770" watchObservedRunningTime="2026-10-30 21:17:03.890900456 +0000 UTC m=+406.813358756"
Oct 30 21:17:04.850984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:04.850956    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d2jcl"
Oct 30 21:17:04.931918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:04.931880    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"]
Oct 30 21:17:04.932212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:04.932132    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" podUID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" containerName="discovery" containerID="cri-o://f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6" gracePeriod=30
Oct 30 21:17:05.177328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.177302    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:17:05.213275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213247    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213302    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213349    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213373    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213412    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213445    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213472    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts\") pod \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\" (UID: \"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b\") "
Oct 30 21:17:05.213928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.213885    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:05.216376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.216335    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:05.216376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.216343    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "kube-api-access-sz442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:05.216570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.216445    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:05.216570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.216460    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:05.217047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.217022    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:05.217669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.217651    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs" pod "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" (UID: "a102033c-8bc4-49b9-9bbc-5c0f28f74a9b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:05.314777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314742    6695 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-cacerts\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.314777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314775    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz442\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-kube-api-access-sz442\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.314777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314786    6695 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-ca-configmap\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.315019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314795    6695 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-kubeconfig\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.315019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314804    6695 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-local-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.315019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314812    6695 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-csr-dns-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.315019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.314820    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:05.854619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.854563    6695 generic.go:356] "Generic (PLEG): container finished" podID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" containerID="f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6" exitCode=0
Oct 30 21:17:05.855037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.854629    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"
Oct 30 21:17:05.855037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.854655    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" event={"ID":"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b","Type":"ContainerDied","Data":"f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6"}
Oct 30 21:17:05.855037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.854712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj" event={"ID":"a102033c-8bc4-49b9-9bbc-5c0f28f74a9b","Type":"ContainerDied","Data":"4233de7f5fab543f8541b1d5f7e08003a430fb4d327872a3d5fbffba1c099483"}
Oct 30 21:17:05.855037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.854739    6695 scope.go:122] "RemoveContainer" containerID="f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6"
Oct 30 21:17:05.867287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.867261    6695 scope.go:122] "RemoveContainer" containerID="f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6"
Oct 30 21:17:05.867577 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:17:05.867555    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6\": container with ID starting with f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6 not found: ID does not exist" containerID="f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6"
Oct 30 21:17:05.867654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.867587    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6"} err="failed to get container status \"f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6\": rpc error: code = NotFound desc = could not find container \"f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6\": container with ID starting with f8813fe95df00a6a7cadd8e19baf00faa8271dbc8eff41fb05992ed5b764b2c6 not found: ID does not exist"
Oct 30 21:17:05.917475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.917432    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"]
Oct 30 21:17:05.928921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:05.928889    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-895pj"]
Oct 30 21:17:07.212231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:07.212194    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" path="/var/lib/kubelet/pods/a102033c-8bc4-49b9-9bbc-5c0f28f74a9b/volumes"
Oct 30 21:17:12.444631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.444575    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d5d9d57cb-vm9fs"]
Oct 30 21:17:12.446304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.446271    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" containerName="discovery"
Oct 30 21:17:12.446304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.446298    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" containerName="discovery"
Oct 30 21:17:12.446593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.446579    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a102033c-8bc4-49b9-9bbc-5c0f28f74a9b" containerName="discovery"
Oct 30 21:17:12.453056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.453023    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.480120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.480076    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:17:12.480120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.480085    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:17:12.480335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.480209    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:17:12.480335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.480227    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cb9nw\""
Oct 30 21:17:12.481176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.481155    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d5d9d57cb-vm9fs"]
Oct 30 21:17:12.576947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.576909    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwts\" (UniqueName: \"kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.576947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.576958    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.577220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.577077    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.678422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.678386    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.678585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.678468    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.678585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.678508    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwts\" (UniqueName: \"kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.681003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.680970    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.681128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.681001    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.696342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.696277    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwts\" (UniqueName: \"kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts\") pod \"maas-controller-7d5d9d57cb-vm9fs\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") " pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.765715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.765639    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:12.969483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.967288    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9484675-8227r"]
Oct 30 21:17:12.993347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.993307    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9484675-8227r"]
Oct 30 21:17:12.993347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:12.993351    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.082996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.082954    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-metrics-certs\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.083184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.083031    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-webhook-cert\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.083184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.083120    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqdb\" (UniqueName: \"kubernetes.io/projected/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-kube-api-access-xqqdb\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.095987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.095942    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d5d9d57cb-vm9fs"]
Oct 30 21:17:13.102278 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:17:13.102247    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0469730b_6c1a_40d1_9115_8c31c52c62ca.slice/crio-04fa277c0cb2699aa907dcdf50b8ad42d31104f8a4f343f211af56db4f44c4ee WatchSource:0}: Error finding container 04fa277c0cb2699aa907dcdf50b8ad42d31104f8a4f343f211af56db4f44c4ee: Status 404 returned error can't find the container with id 04fa277c0cb2699aa907dcdf50b8ad42d31104f8a4f343f211af56db4f44c4ee
Oct 30 21:17:13.184280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.184241    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-metrics-certs\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.184496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.184311    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-webhook-cert\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.184496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.184354    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqdb\" (UniqueName: \"kubernetes.io/projected/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-kube-api-access-xqqdb\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.186729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.186702    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-metrics-certs\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.186836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.186745    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-webhook-cert\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.204421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.204392    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqdb\" (UniqueName: \"kubernetes.io/projected/e7f1f212-3b3e-4475-9d1f-2069f3f7002e-kube-api-access-xqqdb\") pod \"maas-controller-7b9484675-8227r\" (UID: \"e7f1f212-3b3e-4475-9d1f-2069f3f7002e\") " pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.328172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.328118    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:13.627192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.627099    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9484675-8227r"]
Oct 30 21:17:13.631804 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:17:13.631766    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f1f212_3b3e_4475_9d1f_2069f3f7002e.slice/crio-7b45800f448f971e5795590d88135fe1897b1400a0df4976f9c24ff08d45c183 WatchSource:0}: Error finding container 7b45800f448f971e5795590d88135fe1897b1400a0df4976f9c24ff08d45c183: Status 404 returned error can't find the container with id 7b45800f448f971e5795590d88135fe1897b1400a0df4976f9c24ff08d45c183
Oct 30 21:17:13.892697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.892594    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs" event={"ID":"0469730b-6c1a-40d1-9115-8c31c52c62ca","Type":"ContainerStarted","Data":"04fa277c0cb2699aa907dcdf50b8ad42d31104f8a4f343f211af56db4f44c4ee"}
Oct 30 21:17:13.894267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:13.894229    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9484675-8227r" event={"ID":"e7f1f212-3b3e-4475-9d1f-2069f3f7002e","Type":"ContainerStarted","Data":"7b45800f448f971e5795590d88135fe1897b1400a0df4976f9c24ff08d45c183"}
Oct 30 21:17:15.908129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:15.908082    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs" event={"ID":"0469730b-6c1a-40d1-9115-8c31c52c62ca","Type":"ContainerStarted","Data":"3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107"}
Oct 30 21:17:15.908621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:15.908213    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:15.910782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:15.910712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9484675-8227r" event={"ID":"e7f1f212-3b3e-4475-9d1f-2069f3f7002e","Type":"ContainerStarted","Data":"68656a70e75fc118ccefa34d6aae43560979313cae94a446b0d3865c10265e09"}
Oct 30 21:17:15.910931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:15.910846    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:15.952249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:15.952185    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs" podStartSLOduration=1.243595142 podStartE2EDuration="3.952165086s" totalImagesPullingTime="2.708569944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:15.951140029 +0000 UTC m=+418.873598337" watchObservedRunningTime="2026-10-30 21:17:15.952165086 +0000 UTC m=+418.874623383"
Oct 30 21:17:15.990373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:15.990315    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9484675-8227r" podStartSLOduration=1.8016686960000001 podStartE2EDuration="3.9902999s" totalImagesPullingTime="2.188631204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:15.989642889 +0000 UTC m=+418.912101187" watchObservedRunningTime="2026-10-30 21:17:15.9902999 +0000 UTC m=+418.912758192"
Oct 30 21:17:17.869684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.869637    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"]
Oct 30 21:17:17.874307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.874284    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:17.891465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.891382    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"]
Oct 30 21:17:17.900440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.900408    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 21:17:17.900440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.900415    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 21:17:17.900667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.900471    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 21:17:17.991928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:17.991899    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-xwzdq\""
Oct 30 21:17:18.037046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.037012    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss7wl\" (UniqueName: \"kubernetes.io/projected/c755a4aa-1eff-462d-9df5-ef8365fe69ae-kube-api-access-ss7wl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.037249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.037061    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c755a4aa-1eff-462d-9df5-ef8365fe69ae-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.037249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.037153    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c755a4aa-1eff-462d-9df5-ef8365fe69ae-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.037249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.037213    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c755a4aa-1eff-462d-9df5-ef8365fe69ae-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.138563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.138461    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c755a4aa-1eff-462d-9df5-ef8365fe69ae-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.138563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.138533    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c755a4aa-1eff-462d-9df5-ef8365fe69ae-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.138840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.138573    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c755a4aa-1eff-462d-9df5-ef8365fe69ae-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.138840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.138661    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss7wl\" (UniqueName: \"kubernetes.io/projected/c755a4aa-1eff-462d-9df5-ef8365fe69ae-kube-api-access-ss7wl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.139314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.139285    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c755a4aa-1eff-462d-9df5-ef8365fe69ae-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.139497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.139479    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c755a4aa-1eff-462d-9df5-ef8365fe69ae-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.141237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.141214    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c755a4aa-1eff-462d-9df5-ef8365fe69ae-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.159692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.159650    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss7wl\" (UniqueName: \"kubernetes.io/projected/c755a4aa-1eff-462d-9df5-ef8365fe69ae-kube-api-access-ss7wl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-z7b58\" (UID: \"c755a4aa-1eff-462d-9df5-ef8365fe69ae\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.187550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.187513    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:18.491458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.491354    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"]
Oct 30 21:17:18.497070 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:17:18.497037    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc755a4aa_1eff_462d_9df5_ef8365fe69ae.slice/crio-fdaa18d94e13543287184bea8a3d3b45d0c7ce2c6d3bad66c8b276d93af4f760 WatchSource:0}: Error finding container fdaa18d94e13543287184bea8a3d3b45d0c7ce2c6d3bad66c8b276d93af4f760: Status 404 returned error can't find the container with id fdaa18d94e13543287184bea8a3d3b45d0c7ce2c6d3bad66c8b276d93af4f760
Oct 30 21:17:18.924836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:18.924794    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58" event={"ID":"c755a4aa-1eff-462d-9df5-ef8365fe69ae","Type":"ContainerStarted","Data":"fdaa18d94e13543287184bea8a3d3b45d0c7ce2c6d3bad66c8b276d93af4f760"}
Oct 30 21:17:19.561117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.561081    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"]
Oct 30 21:17:19.566009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.565982    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.592065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.592030    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:17:19.592065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.592054    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:17:19.592352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.592082    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:17:19.592352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.592113    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:17:19.592352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.592132    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-5755h\""
Oct 30 21:17:19.592352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.592165    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:17:19.593229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.593211    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"]
Oct 30 21:17:19.652081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.652043    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d8693c-9b0f-4ec9-919c-01933d9b4f19-metrics-certs\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.652268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.652122    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzzt\" (UniqueName: \"kubernetes.io/projected/96d8693c-9b0f-4ec9-919c-01933d9b4f19-kube-api-access-qkzzt\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.652268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.652244    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96d8693c-9b0f-4ec9-919c-01933d9b4f19-cert\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.652414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.652300    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/96d8693c-9b0f-4ec9-919c-01933d9b4f19-manager-config\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.754928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.753277    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d8693c-9b0f-4ec9-919c-01933d9b4f19-metrics-certs\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.754928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.753355    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzzt\" (UniqueName: \"kubernetes.io/projected/96d8693c-9b0f-4ec9-919c-01933d9b4f19-kube-api-access-qkzzt\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.754928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.753394    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96d8693c-9b0f-4ec9-919c-01933d9b4f19-cert\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.754928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.753421    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/96d8693c-9b0f-4ec9-919c-01933d9b4f19-manager-config\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.754928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.754194    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/96d8693c-9b0f-4ec9-919c-01933d9b4f19-manager-config\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.759007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.758981    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d8693c-9b0f-4ec9-919c-01933d9b4f19-metrics-certs\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.759180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.759153    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96d8693c-9b0f-4ec9-919c-01933d9b4f19-cert\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.786594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.786555    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzzt\" (UniqueName: \"kubernetes.io/projected/96d8693c-9b0f-4ec9-919c-01933d9b4f19-kube-api-access-qkzzt\") pod \"jobset-controller-manager-6774f96f6-v2lpn\" (UID: \"96d8693c-9b0f-4ec9-919c-01933d9b4f19\") " pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:19.879504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:19.879412    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:20.225006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:20.224920    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"]
Oct 30 21:17:20.233329 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:17:20.233276    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d8693c_9b0f_4ec9_919c_01933d9b4f19.slice/crio-38e96fae51b77e8167a0050a489e8b4c01fb9e254bf3106da5a8e6203623f83b WatchSource:0}: Error finding container 38e96fae51b77e8167a0050a489e8b4c01fb9e254bf3106da5a8e6203623f83b: Status 404 returned error can't find the container with id 38e96fae51b77e8167a0050a489e8b4c01fb9e254bf3106da5a8e6203623f83b
Oct 30 21:17:20.940991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:20.940947    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn" event={"ID":"96d8693c-9b0f-4ec9-919c-01933d9b4f19","Type":"ContainerStarted","Data":"38e96fae51b77e8167a0050a489e8b4c01fb9e254bf3106da5a8e6203623f83b"}
Oct 30 21:17:23.969994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:23.969949    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn" event={"ID":"96d8693c-9b0f-4ec9-919c-01933d9b4f19","Type":"ContainerStarted","Data":"9ed89081631241e0471d7b6891c6b368dde5e578c6ea9815b092e0554ba384a4"}
Oct 30 21:17:23.970432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:23.970164    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:26.918036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:26.918001    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:26.918477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:26.918096    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9484675-8227r"
Oct 30 21:17:26.958282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:26.958226    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn" podStartSLOduration=4.352295849 podStartE2EDuration="7.958205184s" totalImagesPullingTime="3.605909335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:24.008905284 +0000 UTC m=+426.931363582" watchObservedRunningTime="2026-10-30 21:17:26.958205184 +0000 UTC m=+429.880663485"
Oct 30 21:17:27.022149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.022108    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d5d9d57cb-vm9fs"]
Oct 30 21:17:27.022713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.022343    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs" podUID="0469730b-6c1a-40d1-9115-8c31c52c62ca" containerName="manager" containerID="cri-o://3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107" gracePeriod=30
Oct 30 21:17:27.285578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.285550    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:27.437957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.437908    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts\" (UniqueName: \"kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts\") pod \"0469730b-6c1a-40d1-9115-8c31c52c62ca\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") "
Oct 30 21:17:27.438132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.438060    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert\") pod \"0469730b-6c1a-40d1-9115-8c31c52c62ca\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") "
Oct 30 21:17:27.438132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.438088    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs\") pod \"0469730b-6c1a-40d1-9115-8c31c52c62ca\" (UID: \"0469730b-6c1a-40d1-9115-8c31c52c62ca\") "
Oct 30 21:17:27.441168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.441121    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs" pod "0469730b-6c1a-40d1-9115-8c31c52c62ca" (UID: "0469730b-6c1a-40d1-9115-8c31c52c62ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:27.441302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.441225    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts" pod "0469730b-6c1a-40d1-9115-8c31c52c62ca" (UID: "0469730b-6c1a-40d1-9115-8c31c52c62ca"). InnerVolumeSpecName "kube-api-access-lpwts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:27.441302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.441225    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert" pod "0469730b-6c1a-40d1-9115-8c31c52c62ca" (UID: "0469730b-6c1a-40d1-9115-8c31c52c62ca"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:27.539153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.539108    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwts\" (UniqueName: \"kubernetes.io/projected/0469730b-6c1a-40d1-9115-8c31c52c62ca-kube-api-access-lpwts\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:27.539153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.539151    6695 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-webhook-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:27.539394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.539166    6695 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0469730b-6c1a-40d1-9115-8c31c52c62ca-metrics-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:17:27.991532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.991447    6695 generic.go:356] "Generic (PLEG): container finished" podID="0469730b-6c1a-40d1-9115-8c31c52c62ca" containerID="3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107" exitCode=0
Oct 30 21:17:27.991532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.991512    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs"
Oct 30 21:17:27.992214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.991598    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs" event={"ID":"0469730b-6c1a-40d1-9115-8c31c52c62ca","Type":"ContainerDied","Data":"3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107"}
Oct 30 21:17:27.992214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.991688    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d5d9d57cb-vm9fs" event={"ID":"0469730b-6c1a-40d1-9115-8c31c52c62ca","Type":"ContainerDied","Data":"04fa277c0cb2699aa907dcdf50b8ad42d31104f8a4f343f211af56db4f44c4ee"}
Oct 30 21:17:27.992214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:27.991715    6695 scope.go:122] "RemoveContainer" containerID="3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107"
Oct 30 21:17:28.003637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:28.003501    6695 scope.go:122] "RemoveContainer" containerID="3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107"
Oct 30 21:17:28.004429 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:17:28.004403    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107\": container with ID starting with 3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107 not found: ID does not exist" containerID="3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107"
Oct 30 21:17:28.004530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:28.004438    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107"} err="failed to get container status \"3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107\": rpc error: code = NotFound desc = could not find container \"3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107\": container with ID starting with 3d60a3bc3bbbb740c9e785432731e029720c9ac4f134e10e2667cb58f15bf107 not found: ID does not exist"
Oct 30 21:17:28.040804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:28.040724    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d5d9d57cb-vm9fs"]
Oct 30 21:17:28.051893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:28.051859    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d5d9d57cb-vm9fs"]
Oct 30 21:17:29.212524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:29.212486    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0469730b-6c1a-40d1-9115-8c31c52c62ca" path="/var/lib/kubelet/pods/0469730b-6c1a-40d1-9115-8c31c52c62ca/volumes"
Oct 30 21:17:30.003152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:30.003110    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58" event={"ID":"c755a4aa-1eff-462d-9df5-ef8365fe69ae","Type":"ContainerStarted","Data":"0a4ca381c39dd827577d976bd1b6a26f72704c2b3b51cbe81be00c4fda281e21"}
Oct 30 21:17:30.003343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:30.003265    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:17:30.041380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:30.041331    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58" podStartSLOduration=2.205427767 podStartE2EDuration="13.041317861s" totalImagesPullingTime="10.835890094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:30.0409433 +0000 UTC m=+432.963401599" watchObservedRunningTime="2026-10-30 21:17:30.041317861 +0000 UTC m=+432.963776160"
Oct 30 21:17:31.072686 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:17:31.072649    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:17:34.980921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:34.980883    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6774f96f6-v2lpn"
Oct 30 21:17:41.014021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:17:41.013988    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-z7b58"
Oct 30 21:18:00.248049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.247965    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bvpwb"]
Oct 30 21:18:00.248519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.248445    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0469730b-6c1a-40d1-9115-8c31c52c62ca" containerName="manager"
Oct 30 21:18:00.248519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.248462    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0469730b-6c1a-40d1-9115-8c31c52c62ca" containerName="manager"
Oct 30 21:18:00.248683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.248568    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0469730b-6c1a-40d1-9115-8c31c52c62ca" containerName="manager"
Oct 30 21:18:00.507479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.507395    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bvpwb"]
Oct 30 21:18:00.507479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.507442    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bvpwb"]
Oct 30 21:18:00.507479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.507462    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q7sfd"]
Oct 30 21:18:00.507709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.507527    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:00.508029 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:18:00.508000    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dftl5], unattached volumes=[], failed to process volumes=[kube-api-access-dftl5]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-bvpwb" podUID="1f6a47be-b200-4e47-9e1e-998ec8396516"
Oct 30 21:18:00.538524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.538477    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q7sfd"]
Oct 30 21:18:00.538740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.538625    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.572006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.571975    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:18:00.647084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.647040    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert\") pod \"authorino-6889cbd8c4-q7sfd\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") " pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.647084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.647088    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhtp\" (UniqueName: \"kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp\") pod \"authorino-6889cbd8c4-q7sfd\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") " pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.647328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.647251    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftl5\" (UniqueName: \"kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5\") pod \"authorino-6f6944bcdd-bvpwb\" (UID: \"1f6a47be-b200-4e47-9e1e-998ec8396516\") " pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:00.650459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.650425    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q7sfd"]
Oct 30 21:18:00.650753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:18:00.650718    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xkhtp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-q7sfd" podUID="ca59943d-8ce5-4651-b468-a0fa58c1981c"
Oct 30 21:18:00.677297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.677261    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-676bc99d6c-zhcgw"]
Oct 30 21:18:00.685959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.685929    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:00.697714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.697676    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-676bc99d6c-zhcgw"]
Oct 30 21:18:00.748806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.748755    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftl5\" (UniqueName: \"kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5\") pod \"authorino-6f6944bcdd-bvpwb\" (UID: \"1f6a47be-b200-4e47-9e1e-998ec8396516\") " pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:00.749021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.748817    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert\") pod \"authorino-6889cbd8c4-q7sfd\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") " pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.749021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.748841    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert\") pod \"authorino-676bc99d6c-zhcgw\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") " pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:00.749021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.748858    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhtp\" (UniqueName: \"kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp\") pod \"authorino-6889cbd8c4-q7sfd\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") " pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.749021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.748876    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh27x\" (UniqueName: \"kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x\") pod \"authorino-676bc99d6c-zhcgw\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") " pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:00.751385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.751358    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert\") pod \"authorino-6889cbd8c4-q7sfd\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") " pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.767588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.767527    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftl5\" (UniqueName: \"kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5\") pod \"authorino-6f6944bcdd-bvpwb\" (UID: \"1f6a47be-b200-4e47-9e1e-998ec8396516\") " pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:00.767757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.767732    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhtp\" (UniqueName: \"kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp\") pod \"authorino-6889cbd8c4-q7sfd\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") " pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:00.850033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.849989    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert\") pod \"authorino-676bc99d6c-zhcgw\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") " pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:00.850033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.850035    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh27x\" (UniqueName: \"kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x\") pod \"authorino-676bc99d6c-zhcgw\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") " pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:00.852560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.852524    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert\") pod \"authorino-676bc99d6c-zhcgw\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") " pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:00.868578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:00.868544    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh27x\" (UniqueName: \"kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x\") pod \"authorino-676bc99d6c-zhcgw\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") " pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:01.006723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.006686    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:18:01.084654 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:18:01.084594    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:18:01.135774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.135735    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:01.135948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.135735    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:01.145646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.145616    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:01.148224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.148202    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:01.253386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.253351    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert\" (UniqueName: \"kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert\") pod \"ca59943d-8ce5-4651-b468-a0fa58c1981c\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") "
Oct 30 21:18:01.253906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.253494    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp\" (UniqueName: \"kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp\") pod \"ca59943d-8ce5-4651-b468-a0fa58c1981c\" (UID: \"ca59943d-8ce5-4651-b468-a0fa58c1981c\") "
Oct 30 21:18:01.253906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.253526    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5\" (UniqueName: \"kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5\") pod \"1f6a47be-b200-4e47-9e1e-998ec8396516\" (UID: \"1f6a47be-b200-4e47-9e1e-998ec8396516\") "
Oct 30 21:18:01.255735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.255707    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert" pod "ca59943d-8ce5-4651-b468-a0fa58c1981c" (UID: "ca59943d-8ce5-4651-b468-a0fa58c1981c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:01.255833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.255711    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp" pod "ca59943d-8ce5-4651-b468-a0fa58c1981c" (UID: "ca59943d-8ce5-4651-b468-a0fa58c1981c"). InnerVolumeSpecName "kube-api-access-xkhtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:01.255833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.255717    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5" pod "1f6a47be-b200-4e47-9e1e-998ec8396516" (UID: "1f6a47be-b200-4e47-9e1e-998ec8396516"). InnerVolumeSpecName "kube-api-access-dftl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:01.319146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.319058    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-676bc99d6c-zhcgw"]
Oct 30 21:18:01.324797 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:18:01.324763    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97453484_7ff4_45b9_b80d_92700e1330bd.slice/crio-5d6a42d178baadf97e447f4f7f3c5f6913a92b044bddf9a731985f4253c72a9a WatchSource:0}: Error finding container 5d6a42d178baadf97e447f4f7f3c5f6913a92b044bddf9a731985f4253c72a9a: Status 404 returned error can't find the container with id 5d6a42d178baadf97e447f4f7f3c5f6913a92b044bddf9a731985f4253c72a9a
Oct 30 21:18:01.355129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.355096    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhtp\" (UniqueName: \"kubernetes.io/projected/ca59943d-8ce5-4651-b468-a0fa58c1981c-kube-api-access-xkhtp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:18:01.355129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.355123    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dftl5\" (UniqueName: \"kubernetes.io/projected/1f6a47be-b200-4e47-9e1e-998ec8396516-kube-api-access-dftl5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:18:01.355129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:01.355132    6695 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca59943d-8ce5-4651-b468-a0fa58c1981c-tls-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:18:02.141692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.141637    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-676bc99d6c-zhcgw" event={"ID":"97453484-7ff4-45b9-b80d-92700e1330bd","Type":"ContainerStarted","Data":"5d6a42d178baadf97e447f4f7f3c5f6913a92b044bddf9a731985f4253c72a9a"}
Oct 30 21:18:02.141692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.141684    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bvpwb"
Oct 30 21:18:02.141919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.141683    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q7sfd"
Oct 30 21:18:02.225943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.225902    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q7sfd"]
Oct 30 21:18:02.229738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.229692    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q7sfd"]
Oct 30 21:18:02.282326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.282288    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bvpwb"]
Oct 30 21:18:02.285763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:02.285733    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bvpwb"]
Oct 30 21:18:03.148765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.148725    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-676bc99d6c-zhcgw" event={"ID":"97453484-7ff4-45b9-b80d-92700e1330bd","Type":"ContainerStarted","Data":"9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6"}
Oct 30 21:18:03.185815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.185754    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-676bc99d6c-zhcgw" podStartSLOduration=1.865035265 podStartE2EDuration="3.185739115s" totalImagesPullingTime="1.32070385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:03.184943498 +0000 UTC m=+466.107401796" watchObservedRunningTime="2026-10-30 21:18:03.185739115 +0000 UTC m=+466.108197413"
Oct 30 21:18:03.221349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.221308    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6a47be-b200-4e47-9e1e-998ec8396516" path="/var/lib/kubelet/pods/1f6a47be-b200-4e47-9e1e-998ec8396516/volumes"
Oct 30 21:18:03.222375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.222349    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca59943d-8ce5-4651-b468-a0fa58c1981c" path="/var/lib/kubelet/pods/ca59943d-8ce5-4651-b468-a0fa58c1981c/volumes"
Oct 30 21:18:03.222843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.222822    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r9qb2"]
Oct 30 21:18:03.223203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.223179    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2" podUID="4e6040aa-b37a-4715-b9f6-74cffeef487c" containerName="authorino" containerID="cri-o://d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e" gracePeriod=30
Oct 30 21:18:03.516029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.515996    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:18:03.580289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.580250    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb\" (UniqueName: \"kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb\") pod \"4e6040aa-b37a-4715-b9f6-74cffeef487c\" (UID: \"4e6040aa-b37a-4715-b9f6-74cffeef487c\") "
Oct 30 21:18:03.582490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.582457    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb" pod "4e6040aa-b37a-4715-b9f6-74cffeef487c" (UID: "4e6040aa-b37a-4715-b9f6-74cffeef487c"). InnerVolumeSpecName "kube-api-access-k4jpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:03.681663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:03.681536    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jpb\" (UniqueName: \"kubernetes.io/projected/4e6040aa-b37a-4715-b9f6-74cffeef487c-kube-api-access-k4jpb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:18:04.154131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.154099    6695 generic.go:356] "Generic (PLEG): container finished" podID="4e6040aa-b37a-4715-b9f6-74cffeef487c" containerID="d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e" exitCode=0
Oct 30 21:18:04.154330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.154142    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2"
Oct 30 21:18:04.154330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.154176    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2" event={"ID":"4e6040aa-b37a-4715-b9f6-74cffeef487c","Type":"ContainerDied","Data":"d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e"}
Oct 30 21:18:04.154330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.154227    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-r9qb2" event={"ID":"4e6040aa-b37a-4715-b9f6-74cffeef487c","Type":"ContainerDied","Data":"774637cb7139162b73ba73791fdd41666a78401860cfaf8585b35441da171a68"}
Oct 30 21:18:04.154330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.154246    6695 scope.go:122] "RemoveContainer" containerID="d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e"
Oct 30 21:18:04.164220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.164200    6695 scope.go:122] "RemoveContainer" containerID="d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e"
Oct 30 21:18:04.164567 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:18:04.164543    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e\": container with ID starting with d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e not found: ID does not exist" containerID="d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e"
Oct 30 21:18:04.164656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.164578    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e"} err="failed to get container status \"d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e\": rpc error: code = NotFound desc = could not find container \"d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e\": container with ID starting with d25b9abf834c3fac94b067363ed34a68515a2b8ec8b2b318dd7f69e45cf2599e not found: ID does not exist"
Oct 30 21:18:04.201315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.201278    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r9qb2"]
Oct 30 21:18:04.204382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:04.204355    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-r9qb2"]
Oct 30 21:18:05.212177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:18:05.212139    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6040aa-b37a-4715-b9f6-74cffeef487c" path="/var/lib/kubelet/pods/4e6040aa-b37a-4715-b9f6-74cffeef487c/volumes"
Oct 30 21:18:31.072725 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:18:31.072680    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:19:01.073086 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:19:01.073048    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:19:21.762267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.762231    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"]
Oct 30 21:19:21.763677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.763573    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4e6040aa-b37a-4715-b9f6-74cffeef487c" containerName="authorino"
Oct 30 21:19:21.763677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.763598    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6040aa-b37a-4715-b9f6-74cffeef487c" containerName="authorino"
Oct 30 21:19:21.763832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.763701    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6040aa-b37a-4715-b9f6-74cffeef487c" containerName="authorino"
Oct 30 21:19:21.766707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.766686    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.768669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.768642    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583d659a-c61c-4378-a025-f370c0dd1472-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.768773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.768704    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.768773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.768751    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbfxq\" (UniqueName: \"kubernetes.io/projected/583d659a-c61c-4378-a025-f370c0dd1472-kube-api-access-mbfxq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.768923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.768903    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.768997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.768978    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.769051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.769016    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.781114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.781073    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 21:19:21.792376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.792343    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vj8l8\""
Oct 30 21:19:21.792376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.792351    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:19:21.792512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.792412    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:19:21.794549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.794525    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"]
Oct 30 21:19:21.869779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.869736    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583d659a-c61c-4378-a025-f370c0dd1472-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.869976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.869798    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.869976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.869821    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbfxq\" (UniqueName: \"kubernetes.io/projected/583d659a-c61c-4378-a025-f370c0dd1472-kube-api-access-mbfxq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.869976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.869868    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.869976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.869892    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.869976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.869909    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.870288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.870258    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.870390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.870347    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.870390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.870381    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.872148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.872126    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583d659a-c61c-4378-a025-f370c0dd1472-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.872299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.872277    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583d659a-c61c-4378-a025-f370c0dd1472-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:21.888582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:21.888552    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbfxq\" (UniqueName: \"kubernetes.io/projected/583d659a-c61c-4378-a025-f370c0dd1472-kube-api-access-mbfxq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmvr9\" (UID: \"583d659a-c61c-4378-a025-f370c0dd1472\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:22.079516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:22.079477    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:22.379132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:22.379048    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"]
Oct 30 21:19:22.384214 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:22.384188    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583d659a_c61c_4378_a025_f370c0dd1472.slice/crio-0fd159134d2e84578bee35fd4514e16fa1f02a3d9e4c6f522e5fa3f71d77623a WatchSource:0}: Error finding container 0fd159134d2e84578bee35fd4514e16fa1f02a3d9e4c6f522e5fa3f71d77623a: Status 404 returned error can't find the container with id 0fd159134d2e84578bee35fd4514e16fa1f02a3d9e4c6f522e5fa3f71d77623a
Oct 30 21:19:22.502891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:22.502856    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9" event={"ID":"583d659a-c61c-4378-a025-f370c0dd1472","Type":"ContainerStarted","Data":"0fd159134d2e84578bee35fd4514e16fa1f02a3d9e4c6f522e5fa3f71d77623a"}
Oct 30 21:19:23.684530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.681731    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"]
Oct 30 21:19:23.686476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.686447    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.687056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.687025    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.687165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.687078    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grb8v\" (UniqueName: \"kubernetes.io/projected/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-kube-api-access-grb8v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.687221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.687177    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.687266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.687214    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.687393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.687376    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.687439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.687424    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.712403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.712373    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:19:23.714371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.714339    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"]
Oct 30 21:19:23.788372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788340    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788379    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788405    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788436    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb8v\" (UniqueName: \"kubernetes.io/projected/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-kube-api-access-grb8v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788480    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.788920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788878    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.789015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.788921    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.790902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.790874    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.791101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.791085    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:23.807809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:23.807779    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grb8v\" (UniqueName: \"kubernetes.io/projected/b58e7aa0-bbfc-4718-bbe1-cf91a02e2785-kube-api-access-grb8v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7\" (UID: \"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:24.001263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:24.001223    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:24.298516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:24.298483    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"]
Oct 30 21:19:24.303896 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:24.303827    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58e7aa0_bbfc_4718_bbe1_cf91a02e2785.slice/crio-b157e01b119068a18c6eea723eec2056f733a6800ce511d2cb046629f4ca3048 WatchSource:0}: Error finding container b157e01b119068a18c6eea723eec2056f733a6800ce511d2cb046629f4ca3048: Status 404 returned error can't find the container with id b157e01b119068a18c6eea723eec2056f733a6800ce511d2cb046629f4ca3048
Oct 30 21:19:24.513543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:24.513507    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9" event={"ID":"583d659a-c61c-4378-a025-f370c0dd1472","Type":"ContainerStarted","Data":"689859bd0bb2ed27265a1e23d9b395798873006c392eb5900929dbeda180be1f"}
Oct 30 21:19:24.514798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:24.514775    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7" event={"ID":"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785","Type":"ContainerStarted","Data":"b157e01b119068a18c6eea723eec2056f733a6800ce511d2cb046629f4ca3048"}
Oct 30 21:19:24.555234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:24.555141    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9" podStartSLOduration=2.42549395 podStartE2EDuration="3.555124374s" totalImagesPullingTime="1.129630424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:24.554315211 +0000 UTC m=+547.476773526" watchObservedRunningTime="2026-10-30 21:19:24.555124374 +0000 UTC m=+547.477582673"
Oct 30 21:19:25.081626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:25.079711    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:25.521411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:25.521375    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7" event={"ID":"b58e7aa0-bbfc-4718-bbe1-cf91a02e2785","Type":"ContainerStarted","Data":"216e0a865e198f8ee33035ca72484c76f6bec468fc1ce4f6587f30edcc4fecb0"}
Oct 30 21:19:25.563561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:25.563451    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7" podStartSLOduration=2.347946994 podStartE2EDuration="2.563432013s" totalImagesPullingTime="215.485019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:25.561963872 +0000 UTC m=+548.484422171" watchObservedRunningTime="2026-10-30 21:19:25.563432013 +0000 UTC m=+548.485890312"
Oct 30 21:19:26.002040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:26.001943    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:30.127385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.127284    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"]
Oct 30 21:19:30.131205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.131180    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.145723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.145688    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.145921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.145740    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.145921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.145759    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.145921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.145816    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.145921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.145851    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.145921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.145871    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkq5b\" (UniqueName: \"kubernetes.io/projected/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-kube-api-access-wkq5b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.151238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.151052    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"]
Oct 30 21:19:30.159966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.159929    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 21:19:30.247212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247158    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247215    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247326    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247387    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247422    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkq5b\" (UniqueName: \"kubernetes.io/projected/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-kube-api-access-wkq5b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247466    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247680    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247734    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.247789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.247776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.249762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.249728    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.249874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.249844    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.269064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.269040    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkq5b\" (UniqueName: \"kubernetes.io/projected/064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7-kube-api-access-wkq5b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gss94\" (UID: \"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.450140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.450048    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:30.753013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:30.752914    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"]
Oct 30 21:19:30.759496 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:30.759457    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064ef5f5_f2c4_43d8_ad8a_fcbf0b3d8cf7.slice/crio-774447441a49b9b59471e31ca0fb6006a3510f64f7eec5200614a4e1c6de6b2f WatchSource:0}: Error finding container 774447441a49b9b59471e31ca0fb6006a3510f64f7eec5200614a4e1c6de6b2f: Status 404 returned error can't find the container with id 774447441a49b9b59471e31ca0fb6006a3510f64f7eec5200614a4e1c6de6b2f
Oct 30 21:19:31.072911 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:19:31.072807    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:19:31.554751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:31.554712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94" event={"ID":"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7","Type":"ContainerStarted","Data":"4c9d71a5145b3f45aacc2778b72e0b89f753afbf2624f55c93693910c6c3b830"}
Oct 30 21:19:31.555187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:31.554769    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94" event={"ID":"064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7","Type":"ContainerStarted","Data":"774447441a49b9b59471e31ca0fb6006a3510f64f7eec5200614a4e1c6de6b2f"}
Oct 30 21:19:31.592952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:31.592895    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94" podStartSLOduration=1.369115413 podStartE2EDuration="1.592880096s" totalImagesPullingTime="223.764683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:31.59257367 +0000 UTC m=+554.515031968" watchObservedRunningTime="2026-10-30 21:19:31.592880096 +0000 UTC m=+554.515338388"
Oct 30 21:19:32.080422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:32.080384    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:32.092898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:32.092867    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:32.450256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:32.450165    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:32.570810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:32.570775    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9"
Oct 30 21:19:34.005629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:34.003040    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:34.031019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:34.030986    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:34.580955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:34.580919    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7"
Oct 30 21:19:39.284254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.284218    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"]
Oct 30 21:19:39.288488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.288462    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.312138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.312103    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"]
Oct 30 21:19:39.321229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.321198    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c2rf5\""
Oct 30 21:19:39.337236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337201    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337295    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337335    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337422    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337455    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337499    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337540    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkvfc\" (UniqueName: \"kubernetes.io/projected/ec504004-13c4-4d32-882e-a60f4b4d1db5-kube-api-access-fkvfc\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337629    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec504004-13c4-4d32-882e-a60f4b4d1db5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.337727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.337673    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439071    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439106    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439125    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439143    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkvfc\" (UniqueName: \"kubernetes.io/projected/ec504004-13c4-4d32-882e-a60f4b4d1db5-kube-api-access-fkvfc\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439169    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec504004-13c4-4d32-882e-a60f4b4d1db5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439194    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439292    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439326    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439719    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439796    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439872    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.439958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.439879    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.440076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.440058    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec504004-13c4-4d32-882e-a60f4b4d1db5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.442120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.442093    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.442246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.442218    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.458749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.458723    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkvfc\" (UniqueName: \"kubernetes.io/projected/ec504004-13c4-4d32-882e-a60f4b4d1db5-kube-api-access-fkvfc\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.459217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.459188    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec504004-13c4-4d32-882e-a60f4b4d1db5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-k86d4\" (UID: \"ec504004-13c4-4d32-882e-a60f4b4d1db5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.602432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.602344    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:39.916669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:39.916571    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"]
Oct 30 21:19:39.921905 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:39.921876    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec504004_13c4_4d32_882e_a60f4b4d1db5.slice/crio-1c3f17ccd1ded88cfd7c01d0e5f1dbe5b6a64b9d23ef86a6a93744cb0788b83e WatchSource:0}: Error finding container 1c3f17ccd1ded88cfd7c01d0e5f1dbe5b6a64b9d23ef86a6a93744cb0788b83e: Status 404 returned error can't find the container with id 1c3f17ccd1ded88cfd7c01d0e5f1dbe5b6a64b9d23ef86a6a93744cb0788b83e
Oct 30 21:19:40.450261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:40.450228    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:40.462522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:40.462497    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:40.597014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:40.596977    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" event={"ID":"ec504004-13c4-4d32-882e-a60f4b4d1db5","Type":"ContainerStarted","Data":"1c3f17ccd1ded88cfd7c01d0e5f1dbe5b6a64b9d23ef86a6a93744cb0788b83e"}
Oct 30 21:19:40.608555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:40.608529    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gss94"
Oct 30 21:19:42.661659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:42.661628    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:42.661948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:42.661701    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:42.661948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:42.661734    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:43.636853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:43.636809    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" event={"ID":"ec504004-13c4-4d32-882e-a60f4b4d1db5","Type":"ContainerStarted","Data":"f4a4cc39004bd0eaaaad203894e61942dcd8e6270ee06517c199e4c8af0d6e1b"}
Oct 30 21:19:43.701918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:43.701858    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podStartSLOduration=1.964980682 podStartE2EDuration="4.701843158s" totalImagesPullingTime="2.736862476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:43.689440582 +0000 UTC m=+566.611898880" watchObservedRunningTime="2026-10-30 21:19:43.701843158 +0000 UTC m=+566.624301456"
Oct 30 21:19:44.603419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:44.603377    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:44.604871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:44.604844    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:19:44.604998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:44.604896    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 21:19:45.053713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.053671    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c956c655b-c9565"]
Oct 30 21:19:45.058338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.058315    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.075147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.075102    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c956c655b-c9565"]
Oct 30 21:19:45.195996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.195958    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee8f671f-c470-4b14-8873-861278d4ea59-tls-cert\") pod \"authorino-7c956c655b-c9565\" (UID: \"ee8f671f-c470-4b14-8873-861278d4ea59\") " pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.196188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.196026    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhd8\" (UniqueName: \"kubernetes.io/projected/ee8f671f-c470-4b14-8873-861278d4ea59-kube-api-access-lfhd8\") pod \"authorino-7c956c655b-c9565\" (UID: \"ee8f671f-c470-4b14-8873-861278d4ea59\") " pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.297467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.297425    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhd8\" (UniqueName: \"kubernetes.io/projected/ee8f671f-c470-4b14-8873-861278d4ea59-kube-api-access-lfhd8\") pod \"authorino-7c956c655b-c9565\" (UID: \"ee8f671f-c470-4b14-8873-861278d4ea59\") " pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.297698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.297557    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee8f671f-c470-4b14-8873-861278d4ea59-tls-cert\") pod \"authorino-7c956c655b-c9565\" (UID: \"ee8f671f-c470-4b14-8873-861278d4ea59\") " pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.300125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.300097    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee8f671f-c470-4b14-8873-861278d4ea59-tls-cert\") pod \"authorino-7c956c655b-c9565\" (UID: \"ee8f671f-c470-4b14-8873-861278d4ea59\") " pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.336252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.336158    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhd8\" (UniqueName: \"kubernetes.io/projected/ee8f671f-c470-4b14-8873-861278d4ea59-kube-api-access-lfhd8\") pod \"authorino-7c956c655b-c9565\" (UID: \"ee8f671f-c470-4b14-8873-861278d4ea59\") " pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.370772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.370734    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c956c655b-c9565"
Oct 30 21:19:45.603846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.603752    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:19:45.603846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.603809    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 21:19:45.688406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:45.688350    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c956c655b-c9565"]
Oct 30 21:19:45.693084 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:45.693053    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee8f671f_c470_4b14_8873_861278d4ea59.slice/crio-0963f528fd22e1fb82741d4bf8e32c5f22ccd71621af3f34f0f95befbf490f2b WatchSource:0}: Error finding container 0963f528fd22e1fb82741d4bf8e32c5f22ccd71621af3f34f0f95befbf490f2b: Status 404 returned error can't find the container with id 0963f528fd22e1fb82741d4bf8e32c5f22ccd71621af3f34f0f95befbf490f2b
Oct 30 21:19:46.603036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.603004    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:19:46.603453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.603058    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 21:19:46.652411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.652352    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c956c655b-c9565" event={"ID":"ee8f671f-c470-4b14-8873-861278d4ea59","Type":"ContainerStarted","Data":"41247834d7ea97a38c3a8306b6e1862a95be52bc0fd9be8e278698a7a324a527"}
Oct 30 21:19:46.652411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.652398    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c956c655b-c9565" event={"ID":"ee8f671f-c470-4b14-8873-861278d4ea59","Type":"ContainerStarted","Data":"0963f528fd22e1fb82741d4bf8e32c5f22ccd71621af3f34f0f95befbf490f2b"}
Oct 30 21:19:46.688950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.688849    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c956c655b-c9565" podStartSLOduration=0.978754268 podStartE2EDuration="1.688831654s" totalImagesPullingTime="710.077386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:46.688669806 +0000 UTC m=+569.611128107" watchObservedRunningTime="2026-10-30 21:19:46.688831654 +0000 UTC m=+569.611289953"
Oct 30 21:19:46.727626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.725944    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-676bc99d6c-zhcgw"]
Oct 30 21:19:46.727626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:46.726182    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-676bc99d6c-zhcgw" podUID="97453484-7ff4-45b9-b80d-92700e1330bd" containerName="authorino" containerID="cri-o://9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6" gracePeriod=30
Oct 30 21:19:47.055566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.055533    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:19:47.133629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.129942    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert\" (UniqueName: \"kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert\") pod \"97453484-7ff4-45b9-b80d-92700e1330bd\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") "
Oct 30 21:19:47.133629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.130162    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x\" (UniqueName: \"kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x\") pod \"97453484-7ff4-45b9-b80d-92700e1330bd\" (UID: \"97453484-7ff4-45b9-b80d-92700e1330bd\") "
Oct 30 21:19:47.138772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.138705    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x" pod "97453484-7ff4-45b9-b80d-92700e1330bd" (UID: "97453484-7ff4-45b9-b80d-92700e1330bd"). InnerVolumeSpecName "kube-api-access-wh27x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:47.150691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.150652    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert" pod "97453484-7ff4-45b9-b80d-92700e1330bd" (UID: "97453484-7ff4-45b9-b80d-92700e1330bd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:47.233260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.233174    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh27x\" (UniqueName: \"kubernetes.io/projected/97453484-7ff4-45b9-b80d-92700e1330bd-kube-api-access-wh27x\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:19:47.233260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.233222    6695 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97453484-7ff4-45b9-b80d-92700e1330bd-tls-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:19:47.602788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.602750    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:19:47.602974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.602810    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 21:19:47.658092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.657694    6695 generic.go:356] "Generic (PLEG): container finished" podID="97453484-7ff4-45b9-b80d-92700e1330bd" containerID="9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6" exitCode=0
Oct 30 21:19:47.658092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.657737    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676bc99d6c-zhcgw"
Oct 30 21:19:47.658092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.657788    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-676bc99d6c-zhcgw" event={"ID":"97453484-7ff4-45b9-b80d-92700e1330bd","Type":"ContainerDied","Data":"9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6"}
Oct 30 21:19:47.658092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.657826    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-676bc99d6c-zhcgw" event={"ID":"97453484-7ff4-45b9-b80d-92700e1330bd","Type":"ContainerDied","Data":"5d6a42d178baadf97e447f4f7f3c5f6913a92b044bddf9a731985f4253c72a9a"}
Oct 30 21:19:47.658092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.657850    6695 scope.go:122] "RemoveContainer" containerID="9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6"
Oct 30 21:19:47.667967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.667945    6695 scope.go:122] "RemoveContainer" containerID="9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6"
Oct 30 21:19:47.668518 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:19:47.668492    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6\": container with ID starting with 9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6 not found: ID does not exist" containerID="9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6"
Oct 30 21:19:47.668642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.668525    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6"} err="failed to get container status \"9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6\": rpc error: code = NotFound desc = could not find container \"9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6\": container with ID starting with 9de77dad0b2c1a53f0e505d37fddaced375b4eaa681fb253b261ad24d55859e6 not found: ID does not exist"
Oct 30 21:19:47.738282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.737628    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-676bc99d6c-zhcgw"]
Oct 30 21:19:47.738282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:47.737710    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-676bc99d6c-zhcgw"]
Oct 30 21:19:48.603213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:48.603173    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:19:48.603409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:48.603232    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 21:19:49.214017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:49.213976    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97453484-7ff4-45b9-b80d-92700e1330bd" path="/var/lib/kubelet/pods/97453484-7ff4-45b9-b80d-92700e1330bd/volumes"
Oct 30 21:19:49.602886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:49.602830    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:19:49.603069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:49.602901    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 21:19:51.173536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:51.173453    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:51.174147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:51.174121    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:51.175112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:51.175073    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4"
Oct 30 21:19:54.330940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.330895    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"]
Oct 30 21:19:54.331508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.331487    6695 cpu_manager.go:414] "containerMap: removing container" podUID="97453484-7ff4-45b9-b80d-92700e1330bd" containerName="authorino"
Oct 30 21:19:54.331578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.331509    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97453484-7ff4-45b9-b80d-92700e1330bd" containerName="authorino"
Oct 30 21:19:54.331636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.331616    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="97453484-7ff4-45b9-b80d-92700e1330bd" containerName="authorino"
Oct 30 21:19:54.336218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.336193    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.365666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.365631    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"]
Oct 30 21:19:54.371005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.370968    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"]
Oct 30 21:19:54.374766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.374741    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.403724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.403383    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"]
Oct 30 21:19:54.429613 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429554    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.429804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429595    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.429804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429751    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.429934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429896    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.429934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429919    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.430046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429938    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5k5\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.430046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.429953    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.430154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.430074    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.430154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.430125    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.530735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.530696    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.530908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.530743    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqblz\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.530908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.530773    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.530908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.530798    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.530908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.530824    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531035    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531076    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531090    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531112    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531181    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531218    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531247    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531310    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531348    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531372    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531383    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531458    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5k5\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531493    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.531661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531548    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531594    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.531845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.531734    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.532092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.532071    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.532222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.532107    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.533485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.533462    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.533614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.533586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.554138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.552334    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5k5\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.554138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.554080    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gxjrv\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.632750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632657    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqblz\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.632750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632703    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.632750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632722    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.632750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632748    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632766    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632787    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632820    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632869    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.632941    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.633169    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.633238    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.633340    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.633493    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.633707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.633685    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.635129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.635094    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.635231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.635216    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.649697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.649673    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:54.651290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.651261    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.651418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.651370    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqblz\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz\") pod \"maas-default-gateway-openshift-default-77f556d647-xhqfv\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.694112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.694074    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:54.975922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.975834    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"]
Oct 30 21:19:54.980535 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:54.980504    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22c3ca7_17a6_4dbd_9121_94137b194f53.slice/crio-2e040743c66296e7b1387956f627328987ce7e97ee24df7749efa90af9b7f4dc WatchSource:0}: Error finding container 2e040743c66296e7b1387956f627328987ce7e97ee24df7749efa90af9b7f4dc: Status 404 returned error can't find the container with id 2e040743c66296e7b1387956f627328987ce7e97ee24df7749efa90af9b7f4dc
Oct 30 21:19:54.982919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.982889    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:54.982989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.982960    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:54.983025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:54.982994    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:55.019325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.019271    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"]
Oct 30 21:19:55.025250 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:19:55.025209    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81d6aac_e4dc_4f8b_829d_831cf25bfb37.slice/crio-1cc71b5fb729698474c3a377a5c5f6814aecf37c7daaa5ce70a5ac1deebc38f1 WatchSource:0}: Error finding container 1cc71b5fb729698474c3a377a5c5f6814aecf37c7daaa5ce70a5ac1deebc38f1: Status 404 returned error can't find the container with id 1cc71b5fb729698474c3a377a5c5f6814aecf37c7daaa5ce70a5ac1deebc38f1
Oct 30 21:19:55.028113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.028079    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:55.028207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.028148    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:55.028207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.028175    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:55.705885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.705830    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" event={"ID":"b22c3ca7-17a6-4dbd-9121-94137b194f53","Type":"ContainerStarted","Data":"b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22"}
Oct 30 21:19:55.706371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.705906    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" event={"ID":"b22c3ca7-17a6-4dbd-9121-94137b194f53","Type":"ContainerStarted","Data":"2e040743c66296e7b1387956f627328987ce7e97ee24df7749efa90af9b7f4dc"}
Oct 30 21:19:55.707398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.707374    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" event={"ID":"e81d6aac-e4dc-4f8b-829d-831cf25bfb37","Type":"ContainerStarted","Data":"5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301"}
Oct 30 21:19:55.707517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.707423    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" event={"ID":"e81d6aac-e4dc-4f8b-829d-831cf25bfb37","Type":"ContainerStarted","Data":"1cc71b5fb729698474c3a377a5c5f6814aecf37c7daaa5ce70a5ac1deebc38f1"}
Oct 30 21:19:55.785566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:55.785487    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podStartSLOduration=1.785465255 podStartE2EDuration="1.785465255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:55.772399517 +0000 UTC m=+578.694857827" watchObservedRunningTime="2026-10-30 21:19:55.785465255 +0000 UTC m=+578.707923547"
Oct 30 21:19:56.655671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:56.655627    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:19:56.659652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:56.656724    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:19:56.659652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:56.656788    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:19:56.694829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:56.694784    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:19:56.696695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:56.696652    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:19:56.696870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:56.696732    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:19:57.651630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:57.651121    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:19:57.651630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:57.651190    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:19:57.694647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:57.694579    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:19:57.694821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:57.694667    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:19:58.659654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:58.659055    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:19:58.659654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:58.659120    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:19:58.694682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:58.694645    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:19:58.694682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:58.694710    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:19:59.652097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:59.652052    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:19:59.652440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:59.652386    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:19:59.694594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:59.694556    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:19:59.695008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:19:59.694634    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:20:00.655637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:00.655103    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:20:00.655637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:00.655173    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:20:00.695506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:00.695463    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:20:00.695973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:00.695527    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:20:01.083641 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:20:01.083580    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:20:01.651020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:01.650975    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:20:01.651221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:01.651047    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:20:01.694547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:01.694499    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:20:01.694756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:01.694576    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:20:02.650646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:02.650580    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:20:02.651068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:02.650690    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 21:20:02.695526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:02.695478    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:20:02.695770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:02.695535    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 21:20:03.684080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.684039    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:20:03.684590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.684567    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:20:03.685536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.685517    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:20:03.698335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.698313    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:20:03.698643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.698622    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:20:03.699426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.699409    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:20:03.737989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:03.737936    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podStartSLOduration=9.737922791999999 podStartE2EDuration="9.737922792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:55.828457695 +0000 UTC m=+578.750915998" watchObservedRunningTime="2026-10-30 21:20:03.737922792 +0000 UTC m=+586.660381090"
Oct 30 21:20:15.179095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.179055    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"]
Oct 30 21:20:15.189214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.189186    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.201478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.201444    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"]
Oct 30 21:20:15.211056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.211026    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-acb99762-openshift-default-dockercfg-j52k6\""
Oct 30 21:20:15.246447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246412    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246457    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246547    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246628    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246718    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246746    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246772    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246895    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmgzp\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.246966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.246955    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.347979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.347936    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmgzp\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348002    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348036    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348059    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348091    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348141    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348202    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348229    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348262    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348514    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348692    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.348841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.348774    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.349065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.349043    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.350536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.350511    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.350722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.350703    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.366986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.366960    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.367120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.367106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmgzp\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp\") pod \"e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") " pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.508443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.508353    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:15.831796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.831755    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"]
Oct 30 21:20:15.836160 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:15.836123    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741905db_4c68_4b87_ab48_a3c708449e40.slice/crio-2712babc6697480d3cdc40bf05b7b2b1d0815cd0c34960afcad5c75a116f74f6 WatchSource:0}: Error finding container 2712babc6697480d3cdc40bf05b7b2b1d0815cd0c34960afcad5c75a116f74f6: Status 404 returned error can't find the container with id 2712babc6697480d3cdc40bf05b7b2b1d0815cd0c34960afcad5c75a116f74f6
Oct 30 21:20:15.838560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.838525    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:15.838663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.838616    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:15.838663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.838652    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:15.856905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:15.856873    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" event={"ID":"741905db-4c68-4b87-ab48-a3c708449e40","Type":"ContainerStarted","Data":"2712babc6697480d3cdc40bf05b7b2b1d0815cd0c34960afcad5c75a116f74f6"}
Oct 30 21:20:16.864387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:16.864342    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" event={"ID":"741905db-4c68-4b87-ab48-a3c708449e40","Type":"ContainerStarted","Data":"2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"}
Oct 30 21:20:16.931125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:16.931055    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podStartSLOduration=1.931033201 podStartE2EDuration="1.931033201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.918122265 +0000 UTC m=+599.840580564" watchObservedRunningTime="2026-10-30 21:20:16.931033201 +0000 UTC m=+599.853491503"
Oct 30 21:20:17.274179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:17.274151    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:20:17.278001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:17.277977    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:20:17.509164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:17.509128    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:17.514633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:17.514538    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:17.869118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:17.869088    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:17.870008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:17.869987    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:19.666395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.666359    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"]
Oct 30 21:20:19.671148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.671123    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.687664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.687630    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"]
Oct 30 21:20:19.696987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.696958    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bb245e02\""
Oct 30 21:20:19.805887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.805851    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpfj\" (UniqueName: \"kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj\") pod \"payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") " pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.806151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.806131    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume\") pod \"payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") " pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.907195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.907146    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpfj\" (UniqueName: \"kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj\") pod \"payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") " pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.907372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.907204    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume\") pod \"payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") " pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.908000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.907980    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume\") pod \"payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") " pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.928372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.928300    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpfj\" (UniqueName: \"kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj\") pod \"payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") " pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:19.985167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:19.985122    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:20.287207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:20.287137    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"]
Oct 30 21:20:20.291592 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:20.291550    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2730fa_c259_444b_ab43_0685a70d6dc5.slice/crio-16a04bda16a97f9873b7189685c88ced1f2d211b1e18477516573fa87e03a1e9 WatchSource:0}: Error finding container 16a04bda16a97f9873b7189685c88ced1f2d211b1e18477516573fa87e03a1e9: Status 404 returned error can't find the container with id 16a04bda16a97f9873b7189685c88ced1f2d211b1e18477516573fa87e03a1e9
Oct 30 21:20:20.293694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:20.293672    6695 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:20:20.885547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:20.885496    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm" event={"ID":"ef2730fa-c259-444b-ab43-0685a70d6dc5","Type":"ContainerStarted","Data":"16a04bda16a97f9873b7189685c88ced1f2d211b1e18477516573fa87e03a1e9"}
Oct 30 21:20:21.435800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.435753    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"]
Oct 30 21:20:21.440044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.439960    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.457171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.457132    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"]
Oct 30 21:20:21.466794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.466762    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bdcfca55\""
Oct 30 21:20:21.466794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.466762    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bdcfca55-dockercfg-qzgsz\""
Oct 30 21:20:21.515474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.515425    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"]
Oct 30 21:20:21.523208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.523175    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.532136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.531745    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcskp\" (UniqueName: \"kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp\") pod \"payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.532136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.531848    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.595293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.592824    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"]
Oct 30 21:20:21.633257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.633208    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.633423 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.633263    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.633423 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.633392    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxjm\" (UniqueName: \"kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm\") pod \"payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.633561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.633541    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcskp\" (UniqueName: \"kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp\") pod \"payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.634030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.634011    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.652997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.652963    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcskp\" (UniqueName: \"kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp\") pod \"payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.734982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.734897    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.735723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.735038    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxjm\" (UniqueName: \"kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm\") pod \"payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.735846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.735820    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.755262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.755226    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxjm\" (UniqueName: \"kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm\") pod \"payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:21.757903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.757871    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:21.904948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:21.904915    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:22.072530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:22.072365    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"]
Oct 30 21:20:22.077883 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:22.077845    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd9a80c_c1ed_4f9f_a018_e0ed43b23b73.slice/crio-72ec6f427c888f9db14b82f38e64e37fa578c424787f53466256eca468d662fb WatchSource:0}: Error finding container 72ec6f427c888f9db14b82f38e64e37fa578c424787f53466256eca468d662fb: Status 404 returned error can't find the container with id 72ec6f427c888f9db14b82f38e64e37fa578c424787f53466256eca468d662fb
Oct 30 21:20:22.251081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:22.251046    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"]
Oct 30 21:20:22.257067 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:22.257032    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02759ac_23b6_4e4f_a222_a3372dcfbab8.slice/crio-6ee815b05b1bccb6e747397a9aff9d3fa172b6fe9e21bef8754ae5ff0556ac70 WatchSource:0}: Error finding container 6ee815b05b1bccb6e747397a9aff9d3fa172b6fe9e21bef8754ae5ff0556ac70: Status 404 returned error can't find the container with id 6ee815b05b1bccb6e747397a9aff9d3fa172b6fe9e21bef8754ae5ff0556ac70
Oct 30 21:20:22.899162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:22.899131    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm" event={"ID":"b02759ac-23b6-4e4f-a222-a3372dcfbab8","Type":"ContainerStarted","Data":"6ee815b05b1bccb6e747397a9aff9d3fa172b6fe9e21bef8754ae5ff0556ac70"}
Oct 30 21:20:22.901013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:22.900983    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v" event={"ID":"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73","Type":"ContainerStarted","Data":"72ec6f427c888f9db14b82f38e64e37fa578c424787f53466256eca468d662fb"}
Oct 30 21:20:23.342243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.342200    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"]
Oct 30 21:20:23.346295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.346275    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.363868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.363837    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b72f7bd8\""
Oct 30 21:20:23.364027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.363926    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b72f7bd8-dockercfg-xfdbb\""
Oct 30 21:20:23.365448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.365422    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"]
Oct 30 21:20:23.484662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.483641    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.484662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.483846    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fzc\" (UniqueName: \"kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc\") pod \"payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.585291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.585252    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fzc\" (UniqueName: \"kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc\") pod \"payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.585477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.585322    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.586075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.586054    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.605403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.605329    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fzc\" (UniqueName: \"kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc\") pod \"payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.661635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.661569    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:23.907765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.907676    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v" event={"ID":"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73","Type":"ContainerStarted","Data":"573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91"}
Oct 30 21:20:23.907926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.907836    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:23.909341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.909311    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm" event={"ID":"ef2730fa-c259-444b-ab43-0685a70d6dc5","Type":"ContainerStarted","Data":"c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981"}
Oct 30 21:20:23.909485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.909371    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:23.910770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.910750    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm" event={"ID":"b02759ac-23b6-4e4f-a222-a3372dcfbab8","Type":"ContainerStarted","Data":"ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237"}
Oct 30 21:20:23.910871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.910860    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:23.928571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.928535    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"]
Oct 30 21:20:23.933749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.933724    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:23.949143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.949096    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v" podStartSLOduration=1.675468499 podStartE2EDuration="2.949081661s" totalImagesPullingTime="1.273613162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:23.948677384 +0000 UTC m=+606.871135682" watchObservedRunningTime="2026-10-30 21:20:23.949081661 +0000 UTC m=+606.871539958"
Oct 30 21:20:23.954194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.954165    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0a3ffe1c-openshift-default-dockercfg-j8gcv\""
Oct 30 21:20:23.962013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.961982    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"]
Oct 30 21:20:23.967762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.967731    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"]
Oct 30 21:20:23.974015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:23.973982    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45312c58_fec3_45fe_b45d_5259a69c0007.slice/crio-ebc81c4023126b8a30a903cae9e00a28d9cc4d24e12d00ea8476bb2554fa9a7f WatchSource:0}: Error finding container ebc81c4023126b8a30a903cae9e00a28d9cc4d24e12d00ea8476bb2554fa9a7f: Status 404 returned error can't find the container with id ebc81c4023126b8a30a903cae9e00a28d9cc4d24e12d00ea8476bb2554fa9a7f
Oct 30 21:20:23.985832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:23.985771    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm" podStartSLOduration=2.384794873 podStartE2EDuration="4.985753825s" totalImagesPullingTime="2.600958952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:23.985159523 +0000 UTC m=+606.907617822" watchObservedRunningTime="2026-10-30 21:20:23.985753825 +0000 UTC m=+606.908212127"
Oct 30 21:20:24.063731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.063673    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm" podStartSLOduration=2.42757594 podStartE2EDuration="3.063658526s" totalImagesPullingTime="636.082586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:24.062189552 +0000 UTC m=+606.984647850" watchObservedRunningTime="2026-10-30 21:20:24.063658526 +0000 UTC m=+606.986116818"
Oct 30 21:20:24.091885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.091848    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.091885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.091883    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdq4\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092076    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092148    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092286    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092417    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092470    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092489    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.092515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.092507    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.193811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193714    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.193811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193785    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193821    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193872    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193895    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193922    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193974    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.193996    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdq4\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.194053    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.194394    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.194559    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.194573    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.194872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.194848    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.195103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.195084    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.196899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.196875    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.197699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.197639    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.216893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.216261    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.217227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.217175    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdq4\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4\") pod \"e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") " pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.249846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.249811    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:24.568117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.568083    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"]
Oct 30 21:20:24.575787 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:24.575709    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7dae51_165f_4ba4_93b3_979c00d3e0e5.slice/crio-3c039767c9be7aca62ecb80fe9f5e778b251b3b2695952b1e89d9a3ffb9af525 WatchSource:0}: Error finding container 3c039767c9be7aca62ecb80fe9f5e778b251b3b2695952b1e89d9a3ffb9af525: Status 404 returned error can't find the container with id 3c039767c9be7aca62ecb80fe9f5e778b251b3b2695952b1e89d9a3ffb9af525
Oct 30 21:20:24.577598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.577565    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:24.577704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.577653    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:24.577704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.577683    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:24.917992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.917910    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" event={"ID":"4d7dae51-165f-4ba4-93b3-979c00d3e0e5","Type":"ContainerStarted","Data":"f48a3d939ef03fcec6c06c9b3f922d7a4d7de39e949c44e2721898a9ffe47a34"}
Oct 30 21:20:24.917992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.917972    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" event={"ID":"4d7dae51-165f-4ba4-93b3-979c00d3e0e5","Type":"ContainerStarted","Data":"3c039767c9be7aca62ecb80fe9f5e778b251b3b2695952b1e89d9a3ffb9af525"}
Oct 30 21:20:24.919819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.919789    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr" event={"ID":"45312c58-fec3-45fe-b45d-5259a69c0007","Type":"ContainerStarted","Data":"64c6f4d09383b9965dbd6349d8151eb827a55c8f5f53164ff89b9ac9e20dbec9"}
Oct 30 21:20:24.920004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.919983    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr" event={"ID":"45312c58-fec3-45fe-b45d-5259a69c0007","Type":"ContainerStarted","Data":"ebc81c4023126b8a30a903cae9e00a28d9cc4d24e12d00ea8476bb2554fa9a7f"}
Oct 30 21:20:24.920142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.920129    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:24.960558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.960505    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podStartSLOduration=1.9604853009999998 podStartE2EDuration="1.960485301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:24.959102859 +0000 UTC m=+607.881561160" watchObservedRunningTime="2026-10-30 21:20:24.960485301 +0000 UTC m=+607.882943611"
Oct 30 21:20:24.995934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:24.995884    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr" podStartSLOduration=1.995868599 podStartE2EDuration="1.995868599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:24.99541476 +0000 UTC m=+607.917873059" watchObservedRunningTime="2026-10-30 21:20:24.995868599 +0000 UTC m=+607.918326898"
Oct 30 21:20:25.250407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:25.250321    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:25.256817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:25.256782    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:25.925101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:25.925062    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:25.925969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:25.925951    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:28.899043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.899002    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"]
Oct 30 21:20:28.911569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.911542    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:28.922734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.922705    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"]
Oct 30 21:20:28.931901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.931874    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0a3ffe1c\""
Oct 30 21:20:28.939582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.939561    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0a3ffe1c-dockercfg-rkfwc\""
Oct 30 21:20:28.958340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.958305    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:28.958483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:28.958447    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr\") pod \"payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.020402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.020366    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"]
Oct 30 21:20:29.033987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.033913    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.058171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.058132    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"]
Oct 30 21:20:29.059202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.059169    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume\") pod \"payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") " pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.059340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.059219    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.059432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.059397    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr\") pod \"payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.059543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.059524    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jgf\" (UniqueName: \"kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf\") pod \"payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") " pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.060132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.060106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.079164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.079136    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr\") pod \"payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.160816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.160723    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume\") pod \"payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") " pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.160971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.160870    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jgf\" (UniqueName: \"kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf\") pod \"payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") " pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.161439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.161419    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume\") pod \"payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") " pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.181319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.181290    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jgf\" (UniqueName: \"kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf\") pod \"payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") " pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.227253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.227224    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.362190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.362129    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.524946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.524908    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"]
Oct 30 21:20:29.529293 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:29.529266    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b5ef49_9cda_491f_9eb5_a0f1b333717e.slice/crio-41482e5a63d524da15d4a5d3239103bb64692ebdfee329073e1476e0adb2ee78 WatchSource:0}: Error finding container 41482e5a63d524da15d4a5d3239103bb64692ebdfee329073e1476e0adb2ee78: Status 404 returned error can't find the container with id 41482e5a63d524da15d4a5d3239103bb64692ebdfee329073e1476e0adb2ee78
Oct 30 21:20:29.673271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.673115    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"]
Oct 30 21:20:29.677496 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:29.677468    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d31f1d6_db41_4edc_a453_2de2efc6aa26.slice/crio-ccd1acee0d83eaa99ec536b3d354323eb295eff310925d42b165e41723d4cb2c WatchSource:0}: Error finding container ccd1acee0d83eaa99ec536b3d354323eb295eff310925d42b165e41723d4cb2c: Status 404 returned error can't find the container with id ccd1acee0d83eaa99ec536b3d354323eb295eff310925d42b165e41723d4cb2c
Oct 30 21:20:29.948595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.948508    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp" event={"ID":"4d31f1d6-db41-4edc-a453-2de2efc6aa26","Type":"ContainerStarted","Data":"033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6"}
Oct 30 21:20:29.948595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.948571    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp" event={"ID":"4d31f1d6-db41-4edc-a453-2de2efc6aa26","Type":"ContainerStarted","Data":"ccd1acee0d83eaa99ec536b3d354323eb295eff310925d42b165e41723d4cb2c"}
Oct 30 21:20:29.949108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.948655    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:29.950137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.950114    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr" event={"ID":"43b5ef49-9cda-491f-9eb5-a0f1b333717e","Type":"ContainerStarted","Data":"01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e"}
Oct 30 21:20:29.950293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.950154    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr" event={"ID":"43b5ef49-9cda-491f-9eb5-a0f1b333717e","Type":"ContainerStarted","Data":"41482e5a63d524da15d4a5d3239103bb64692ebdfee329073e1476e0adb2ee78"}
Oct 30 21:20:29.950293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.950232    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:29.986886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:29.986838    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp" podStartSLOduration=1.986823566 podStartE2EDuration="1.986823566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:29.985294843 +0000 UTC m=+612.907753153" watchObservedRunningTime="2026-10-30 21:20:29.986823566 +0000 UTC m=+612.909281864"
Oct 30 21:20:30.023680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:30.023623    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr" podStartSLOduration=2.023591921 podStartE2EDuration="2.023591921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:30.021839068 +0000 UTC m=+612.944297366" watchObservedRunningTime="2026-10-30 21:20:30.023591921 +0000 UTC m=+612.946050218"
Oct 30 21:20:31.073844 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:20:31.073805    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:20:33.837505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.837456    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"]
Oct 30 21:20:33.843118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.843096    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:33.858340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.858311    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"]
Oct 30 21:20:33.867560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.867470    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:20:33.867560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.867506    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:20:33.867841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.867581    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:20:33.869370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.869343    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:20:33.931629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.931233    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f522r\" (UniqueName: \"kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:33.931629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.931398    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:33.931629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:33.931475    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.021629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.017740    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"]
Oct 30 21:20:34.035630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.032227    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f522r\" (UniqueName: \"kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.035630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.032375    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.035630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.032451    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.035630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.033439    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.038942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.037983    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.038942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.038927    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.043036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.043005    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"]
Oct 30 21:20:34.055681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.055637    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f522r\" (UniqueName: \"kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r\") pod \"maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.134690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.133595    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.134690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.133730    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.134690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.133811    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gqq\" (UniqueName: \"kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.157594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.157552    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:34.240280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.240244    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gqq\" (UniqueName: \"kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.240482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.240386    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.240482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.240472    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.242289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.241332    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.246068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.244310    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.263189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.263151    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gqq\" (UniqueName: \"kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq\") pod \"maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.383896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.383860    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:34.486142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.486105    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"]
Oct 30 21:20:34.492100 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:34.492051    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a328bd_8b3e_4cff_8dcd_5ac83e8d7608.slice/crio-8be2923da32a0c02fd33ccf1e6b4e6635b6dc8a7d0dded4ab5be9178852251a8 WatchSource:0}: Error finding container 8be2923da32a0c02fd33ccf1e6b4e6635b6dc8a7d0dded4ab5be9178852251a8: Status 404 returned error can't find the container with id 8be2923da32a0c02fd33ccf1e6b4e6635b6dc8a7d0dded4ab5be9178852251a8
Oct 30 21:20:34.708565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.708472    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"]
Oct 30 21:20:34.937450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.937275    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:34.937450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.937374    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:34.938242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.938217    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:34.988924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.988832    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx" event={"ID":"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608","Type":"ContainerStarted","Data":"8be2923da32a0c02fd33ccf1e6b4e6635b6dc8a7d0dded4ab5be9178852251a8"}
Oct 30 21:20:34.990568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:34.990529    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd" event={"ID":"75c9ebf3-23b1-4491-adb5-1fe9102ebb64","Type":"ContainerStarted","Data":"3c8a4b455672277e96eb9bf809c8cc374ed69b92ccc323d2d9ab87290f84c9ee"}
Oct 30 21:20:35.942718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:35.942683    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:39.020812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:39.020771    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd" event={"ID":"75c9ebf3-23b1-4491-adb5-1fe9102ebb64","Type":"ContainerStarted","Data":"d857b119880051a3cc6701a7697842d8613640bae97d378b52e2e5d9102045cf"}
Oct 30 21:20:39.023397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:39.023361    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx" event={"ID":"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608","Type":"ContainerStarted","Data":"39bd3687c79cc85809e0d12da3787c19db464e7425983525bf73caffaf35b9cb"}
Oct 30 21:20:39.061774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:39.061706    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd" podStartSLOduration=2.5429198570000002 podStartE2EDuration="6.061686172s" totalImagesPullingTime="3.518766315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:39.060359208 +0000 UTC m=+621.982817507" watchObservedRunningTime="2026-10-30 21:20:39.061686172 +0000 UTC m=+621.984144472"
Oct 30 21:20:39.100392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:39.100330    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx" podStartSLOduration=2.330763343 podStartE2EDuration="6.100310159s" totalImagesPullingTime="3.769546816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:39.098093796 +0000 UTC m=+622.020552094" watchObservedRunningTime="2026-10-30 21:20:39.100310159 +0000 UTC m=+622.022768458"
Oct 30 21:20:40.029589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.029545    6695 generic.go:356] "Generic (PLEG): container finished" podID="75c9ebf3-23b1-4491-adb5-1fe9102ebb64" containerID="d857b119880051a3cc6701a7697842d8613640bae97d378b52e2e5d9102045cf" exitCode=0
Oct 30 21:20:40.030087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.029633    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd" event={"ID":"75c9ebf3-23b1-4491-adb5-1fe9102ebb64","Type":"ContainerDied","Data":"d857b119880051a3cc6701a7697842d8613640bae97d378b52e2e5d9102045cf"}
Oct 30 21:20:40.031502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.031468    6695 generic.go:356] "Generic (PLEG): container finished" podID="12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" containerID="39bd3687c79cc85809e0d12da3787c19db464e7425983525bf73caffaf35b9cb" exitCode=0
Oct 30 21:20:40.031622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.031544    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx" event={"ID":"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608","Type":"ContainerDied","Data":"39bd3687c79cc85809e0d12da3787c19db464e7425983525bf73caffaf35b9cb"}
Oct 30 21:20:40.862629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.862571    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"]
Oct 30 21:20:40.866517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.866496    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:40.889475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.889430    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"]
Oct 30 21:20:40.937478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.937441    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:40.937679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.937485    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:40.937745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.937723    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqtt\" (UniqueName: \"kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:40.959070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.959040    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:20:40.959230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:40.959111    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:20:41.039220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.039168    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqtt\" (UniqueName: \"kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.039732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.039448    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.039732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.039486    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.041117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.040239    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.045320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.045295    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.060541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.060498    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqtt\" (UniqueName: \"kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt\") pod \"maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.225025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.223731    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:41.294545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.292844    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:41.302082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.301999    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:41.343259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.343218    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq\" (UniqueName: \"kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq\") pod \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") "
Oct 30 21:20:41.343432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.343362    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca\") pod \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") "
Oct 30 21:20:41.343500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.343465    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca\") pod \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\" (UID: \"75c9ebf3-23b1-4491-adb5-1fe9102ebb64\") "
Oct 30 21:20:41.343557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.343504    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca\") pod \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") "
Oct 30 21:20:41.343557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.343542    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca\") pod \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") "
Oct 30 21:20:41.343678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.343580    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r\" (UniqueName: \"kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r\") pod \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\" (UID: \"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608\") "
Oct 30 21:20:41.346445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.346263    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca" pod "12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" (UID: "12a328bd-8b3e-4cff-8dcd-5ac83e8d7608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:41.346445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.346418    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca" pod "75c9ebf3-23b1-4491-adb5-1fe9102ebb64" (UID: "75c9ebf3-23b1-4491-adb5-1fe9102ebb64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:41.354351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.354288    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca" pod "12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" (UID: "12a328bd-8b3e-4cff-8dcd-5ac83e8d7608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:41.354351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.354288    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r" pod "12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" (UID: "12a328bd-8b3e-4cff-8dcd-5ac83e8d7608"). InnerVolumeSpecName "kube-api-access-f522r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:41.356890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.356840    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq" pod "75c9ebf3-23b1-4491-adb5-1fe9102ebb64" (UID: "75c9ebf3-23b1-4491-adb5-1fe9102ebb64"). InnerVolumeSpecName "kube-api-access-x4gqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:41.357914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.357858    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca" pod "75c9ebf3-23b1-4491-adb5-1fe9102ebb64" (UID: "75c9ebf3-23b1-4491-adb5-1fe9102ebb64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:41.445828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.445648    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gqq\" (UniqueName: \"kubernetes.io/projected/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-kube-api-access-x4gqq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:41.446179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.445688    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:41.446179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.446144    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c9ebf3-23b1-4491-adb5-1fe9102ebb64-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:41.446487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.446162    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:41.446487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.446450    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:41.446487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.446466    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f522r\" (UniqueName: \"kubernetes.io/projected/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608-kube-api-access-f522r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:41.575769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:41.575733    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"]
Oct 30 21:20:41.579059 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:41.579021    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf1c7c9_c4f2_410e_b867_8f763dc15f0e.slice/crio-5097e3f29da7464a00b62ae48e664856362fbe7548a9251b5749ac9f8ebc526c WatchSource:0}: Error finding container 5097e3f29da7464a00b62ae48e664856362fbe7548a9251b5749ac9f8ebc526c: Status 404 returned error can't find the container with id 5097e3f29da7464a00b62ae48e664856362fbe7548a9251b5749ac9f8ebc526c
Oct 30 21:20:42.047476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.047448    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"
Oct 30 21:20:42.048088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.047486    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd" event={"ID":"75c9ebf3-23b1-4491-adb5-1fe9102ebb64","Type":"ContainerDied","Data":"3c8a4b455672277e96eb9bf809c8cc374ed69b92ccc323d2d9ab87290f84c9ee"}
Oct 30 21:20:42.048205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.048124    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8a4b455672277e96eb9bf809c8cc374ed69b92ccc323d2d9ab87290f84c9ee"
Oct 30 21:20:42.049619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.049576    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p" event={"ID":"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e","Type":"ContainerStarted","Data":"ae73ff529239c24610ef4676b500cef6164813a325704d1682b4ef73134675b0"}
Oct 30 21:20:42.049730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.049654    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p" event={"ID":"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e","Type":"ContainerStarted","Data":"5097e3f29da7464a00b62ae48e664856362fbe7548a9251b5749ac9f8ebc526c"}
Oct 30 21:20:42.051188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.051169    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"
Oct 30 21:20:42.052157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.051223    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx" event={"ID":"12a328bd-8b3e-4cff-8dcd-5ac83e8d7608","Type":"ContainerDied","Data":"8be2923da32a0c02fd33ccf1e6b4e6635b6dc8a7d0dded4ab5be9178852251a8"}
Oct 30 21:20:42.052429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.052406    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8be2923da32a0c02fd33ccf1e6b4e6635b6dc8a7d0dded4ab5be9178852251a8"
Oct 30 21:20:42.053981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.053955    6695 generic.go:356] "Generic (PLEG): container finished" podID="741905db-4c68-4b87-ab48-a3c708449e40" containerID="2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100" exitCode=0
Oct 30 21:20:42.054138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.053997    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" event={"ID":"741905db-4c68-4b87-ab48-a3c708449e40","Type":"ContainerDied","Data":"2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"}
Oct 30 21:20:42.054938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.054502    6695 scope.go:122] "RemoveContainer" containerID="2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"
Oct 30 21:20:42.057194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.057165    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:42.057269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.057238    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:42.057306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.057270    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:42.105324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:42.105272    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p" podStartSLOduration=2.105255786 podStartE2EDuration="2.105255786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:42.104075396 +0000 UTC m=+625.026533691" watchObservedRunningTime="2026-10-30 21:20:42.105255786 +0000 UTC m=+625.027714119"
Oct 30 21:20:43.068805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:43.068757    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" event={"ID":"741905db-4c68-4b87-ab48-a3c708449e40","Type":"ContainerStarted","Data":"23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc"}
Oct 30 21:20:43.070584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:43.070557    6695 generic.go:356] "Generic (PLEG): container finished" podID="6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" containerID="ae73ff529239c24610ef4676b500cef6164813a325704d1682b4ef73134675b0" exitCode=0
Oct 30 21:20:43.070745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:43.070671    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p" event={"ID":"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e","Type":"ContainerDied","Data":"ae73ff529239c24610ef4676b500cef6164813a325704d1682b4ef73134675b0"}
Oct 30 21:20:43.508809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:43.508719    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:43.509639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:43.509595    6695 patch_prober.go:29] interesting pod/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:20:43.509741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:43.509661    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:20:44.372642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.372329    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:44.382757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.382728    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt\" (UniqueName: \"kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt\") pod \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") "
Oct 30 21:20:44.382913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.382772    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca\") pod \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") "
Oct 30 21:20:44.382913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.382895    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca\") pod \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\" (UID: \"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e\") "
Oct 30 21:20:44.383750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.383725    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca" pod "6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" (UID: "6cf1c7c9-c4f2-410e-b867-8f763dc15f0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:44.385069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.385036    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca" pod "6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" (UID: "6cf1c7c9-c4f2-410e-b867-8f763dc15f0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:44.385275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.385249    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt" pod "6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" (UID: "6cf1c7c9-c4f2-410e-b867-8f763dc15f0e"). InnerVolumeSpecName "kube-api-access-tjqtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:44.441626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.439020    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"]
Oct 30 21:20:44.472641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.472391    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"]
Oct 30 21:20:44.484179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.484147    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjqtt\" (UniqueName: \"kubernetes.io/projected/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-kube-api-access-tjqtt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:44.484179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.484176    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:44.484179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.484187    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:44.509355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.509318    6695 patch_prober.go:29] interesting pod/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:20:44.509491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.509368    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:20:44.806742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.806708    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"]
Oct 30 21:20:44.807814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.807793    6695 cpu_manager.go:414] "containerMap: removing container" podUID="12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" containerName="revoke-keys"
Oct 30 21:20:44.807980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.807962    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" containerName="revoke-keys"
Oct 30 21:20:44.808104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808093    6695 cpu_manager.go:414] "containerMap: removing container" podUID="75c9ebf3-23b1-4491-adb5-1fe9102ebb64" containerName="revoke-keys"
Oct 30 21:20:44.808184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808173    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c9ebf3-23b1-4491-adb5-1fe9102ebb64" containerName="revoke-keys"
Oct 30 21:20:44.808276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808266    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" containerName="revoke-keys"
Oct 30 21:20:44.808353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808343    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" containerName="revoke-keys"
Oct 30 21:20:44.808614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808589    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" containerName="revoke-keys"
Oct 30 21:20:44.808720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808697    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c9ebf3-23b1-4491-adb5-1fe9102ebb64" containerName="revoke-keys"
Oct 30 21:20:44.808720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.808716    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" containerName="revoke-keys"
Oct 30 21:20:44.821722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.820914    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.832706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.831211    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"]
Oct 30 21:20:44.888981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.888949    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k968p\" (UniqueName: \"kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.888981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.888987    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.889251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.889211    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.990157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.990122    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.990332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.990203    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k968p\" (UniqueName: \"kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.990332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.990233    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.991253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.991227    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:44.993758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:44.992678    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:45.009741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.009711    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k968p\" (UniqueName: \"kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p\") pod \"maas-api-revoke-keys-e2e-ait-acb99762-46xpw\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:45.095268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.095170    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"
Oct 30 21:20:45.095268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.095192    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p" event={"ID":"6cf1c7c9-c4f2-410e-b867-8f763dc15f0e","Type":"ContainerDied","Data":"5097e3f29da7464a00b62ae48e664856362fbe7548a9251b5749ac9f8ebc526c"}
Oct 30 21:20:45.095268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.095240    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5097e3f29da7464a00b62ae48e664856362fbe7548a9251b5749ac9f8ebc526c"
Oct 30 21:20:45.182352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.181822    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:45.509522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.509445    6695 patch_prober.go:29] interesting pod/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:20:45.509522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.509501    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:20:45.546629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.546239    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"]
Oct 30 21:20:45.568930 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:45.568882    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9afa61_0c3d_431c_92b3_5619081096ac.slice/crio-1c9967446445240227d78d0ee669c81ecde672c00dda0ebb532cb7ce6417f5b0 WatchSource:0}: Error finding container 1c9967446445240227d78d0ee669c81ecde672c00dda0ebb532cb7ce6417f5b0: Status 404 returned error can't find the container with id 1c9967446445240227d78d0ee669c81ecde672c00dda0ebb532cb7ce6417f5b0
Oct 30 21:20:45.868623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:45.867710    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"]
Oct 30 21:20:46.102808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:46.102770    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw" event={"ID":"aa9afa61-0c3d-431c-92b3-5619081096ac","Type":"ContainerStarted","Data":"e96fa95ce8e656b5a7c3ea189f3f3bcf4582e8b8f5e396d5c101cb9ec37834c2"}
Oct 30 21:20:46.102985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:46.102833    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw" event={"ID":"aa9afa61-0c3d-431c-92b3-5619081096ac","Type":"ContainerStarted","Data":"1c9967446445240227d78d0ee669c81ecde672c00dda0ebb532cb7ce6417f5b0"}
Oct 30 21:20:46.186947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:46.186886    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw" podStartSLOduration=2.186869189 podStartE2EDuration="2.186869189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:46.18392669 +0000 UTC m=+629.106384982" watchObservedRunningTime="2026-10-30 21:20:46.186869189 +0000 UTC m=+629.109327487"
Oct 30 21:20:46.509770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:46.509676    6695 patch_prober.go:29] interesting pod/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:20:46.509770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:46.509745    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:20:47.109684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:47.109648    6695 generic.go:356] "Generic (PLEG): container finished" podID="aa9afa61-0c3d-431c-92b3-5619081096ac" containerID="e96fa95ce8e656b5a7c3ea189f3f3bcf4582e8b8f5e396d5c101cb9ec37834c2" exitCode=0
Oct 30 21:20:47.109903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:47.109738    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw" event={"ID":"aa9afa61-0c3d-431c-92b3-5619081096ac","Type":"ContainerDied","Data":"e96fa95ce8e656b5a7c3ea189f3f3bcf4582e8b8f5e396d5c101cb9ec37834c2"}
Oct 30 21:20:47.509359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:47.509277    6695 patch_prober.go:29] interesting pod/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:20:47.509359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:47.509336    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:20:47.869191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:47.869154    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:48.297136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.297105    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:48.384544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.384431    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p\" (UniqueName: \"kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p\") pod \"aa9afa61-0c3d-431c-92b3-5619081096ac\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") "
Oct 30 21:20:48.384743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.384680    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca\") pod \"aa9afa61-0c3d-431c-92b3-5619081096ac\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") "
Oct 30 21:20:48.384743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.384733    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca\") pod \"aa9afa61-0c3d-431c-92b3-5619081096ac\" (UID: \"aa9afa61-0c3d-431c-92b3-5619081096ac\") "
Oct 30 21:20:48.385394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.385362    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca" pod "aa9afa61-0c3d-431c-92b3-5619081096ac" (UID: "aa9afa61-0c3d-431c-92b3-5619081096ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:48.387283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.387244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p" pod "aa9afa61-0c3d-431c-92b3-5619081096ac" (UID: "aa9afa61-0c3d-431c-92b3-5619081096ac"). InnerVolumeSpecName "kube-api-access-k968p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:48.387402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.387290    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca" pod "aa9afa61-0c3d-431c-92b3-5619081096ac" (UID: "aa9afa61-0c3d-431c-92b3-5619081096ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:48.486354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.486263    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k968p\" (UniqueName: \"kubernetes.io/projected/aa9afa61-0c3d-431c-92b3-5619081096ac-kube-api-access-k968p\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:48.486354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.486314    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:48.486354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.486327    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9afa61-0c3d-431c-92b3-5619081096ac-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:48.513667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.513638    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:48.514894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:48.514876    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:20:49.133870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.133835    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"
Oct 30 21:20:49.133870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.133853    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw" event={"ID":"aa9afa61-0c3d-431c-92b3-5619081096ac","Type":"ContainerDied","Data":"1c9967446445240227d78d0ee669c81ecde672c00dda0ebb532cb7ce6417f5b0"}
Oct 30 21:20:49.134405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.133906    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c9967446445240227d78d0ee669c81ecde672c00dda0ebb532cb7ce6417f5b0"
Oct 30 21:20:49.439462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.439369    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm" podUID="b02759ac-23b6-4e4f-a222-a3372dcfbab8" containerName="payload-processing" containerID="cri-o://ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237" gracePeriod=25
Oct 30 21:20:49.480548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.480467    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v" podUID="ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" containerName="payload-pre-processing" containerID="cri-o://573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91" gracePeriod=25
Oct 30 21:20:49.530265 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:20:49.530225    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd9a80c_c1ed_4f9f_a018_e0ed43b23b73.slice/crio-573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd9a80c_c1ed_4f9f_a018_e0ed43b23b73.slice/crio-conmon-573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:20:49.800866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.800829    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:49.841728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.841694    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:49.923626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.921426    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm\" (UniqueName: \"kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm\") pod \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") "
Oct 30 21:20:49.923626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.921484    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume\") pod \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") "
Oct 30 21:20:49.923626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.921548    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp\" (UniqueName: \"kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp\") pod \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\" (UID: \"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73\") "
Oct 30 21:20:49.923626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.921584    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume\") pod \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\" (UID: \"b02759ac-23b6-4e4f-a222-a3372dcfbab8\") "
Oct 30 21:20:49.928171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.925418    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm" pod "b02759ac-23b6-4e4f-a222-a3372dcfbab8" (UID: "b02759ac-23b6-4e4f-a222-a3372dcfbab8"). InnerVolumeSpecName "kube-api-access-6pxjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:49.928171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.925540    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume" pod "ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" (UID: "ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:49.928171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.926416    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pxjm\" (UniqueName: \"kubernetes.io/projected/b02759ac-23b6-4e4f-a222-a3372dcfbab8-kube-api-access-6pxjm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:49.928171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.926442    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:49.928171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.926679    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume" pod "b02759ac-23b6-4e4f-a222-a3372dcfbab8" (UID: "b02759ac-23b6-4e4f-a222-a3372dcfbab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:49.928171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:49.927682    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp" pod "ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" (UID: "ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73"). InnerVolumeSpecName "kube-api-access-kcskp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:50.027077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.027031    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcskp\" (UniqueName: \"kubernetes.io/projected/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73-kube-api-access-kcskp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:50.027077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.027062    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02759ac-23b6-4e4f-a222-a3372dcfbab8-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:50.140275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.140234    6695 generic.go:356] "Generic (PLEG): container finished" podID="ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" containerID="573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91" exitCode=0
Oct 30 21:20:50.140275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.140275    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"
Oct 30 21:20:50.140789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.140315    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v" event={"ID":"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73","Type":"ContainerDied","Data":"573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91"}
Oct 30 21:20:50.140789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.140366    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v" event={"ID":"ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73","Type":"ContainerDied","Data":"72ec6f427c888f9db14b82f38e64e37fa578c424787f53466256eca468d662fb"}
Oct 30 21:20:50.140789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.140389    6695 scope.go:122] "RemoveContainer" containerID="573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91"
Oct 30 21:20:50.142106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.142081    6695 generic.go:356] "Generic (PLEG): container finished" podID="b02759ac-23b6-4e4f-a222-a3372dcfbab8" containerID="ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237" exitCode=0
Oct 30 21:20:50.142203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.142127    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"
Oct 30 21:20:50.142203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.142148    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm" event={"ID":"b02759ac-23b6-4e4f-a222-a3372dcfbab8","Type":"ContainerDied","Data":"ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237"}
Oct 30 21:20:50.142203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.142189    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm" event={"ID":"b02759ac-23b6-4e4f-a222-a3372dcfbab8","Type":"ContainerDied","Data":"6ee815b05b1bccb6e747397a9aff9d3fa172b6fe9e21bef8754ae5ff0556ac70"}
Oct 30 21:20:50.152514 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.152496    6695 scope.go:122] "RemoveContainer" containerID="573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91"
Oct 30 21:20:50.152945 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:20:50.152923    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91\": container with ID starting with 573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91 not found: ID does not exist" containerID="573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91"
Oct 30 21:20:50.153020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.152953    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91"} err="failed to get container status \"573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91\": rpc error: code = NotFound desc = could not find container \"573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91\": container with ID starting with 573b4dc4b63f1a0eaeff538d271447dc400f9add32728dc2d9dd689930f25f91 not found: ID does not exist"
Oct 30 21:20:50.153020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.152970    6695 scope.go:122] "RemoveContainer" containerID="ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237"
Oct 30 21:20:50.163197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.163177    6695 scope.go:122] "RemoveContainer" containerID="ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237"
Oct 30 21:20:50.163466 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:20:50.163444    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237\": container with ID starting with ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237 not found: ID does not exist" containerID="ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237"
Oct 30 21:20:50.163530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.163477    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237"} err="failed to get container status \"ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237\": rpc error: code = NotFound desc = could not find container \"ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237\": container with ID starting with ddb27d9a9b2ca25e2729373a5177fc9ff799d1abdd321f8e1406e2289584f237 not found: ID does not exist"
Oct 30 21:20:50.190639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.189465    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"]
Oct 30 21:20:50.202182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.202150    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bdcfca55-5fbff774b7-vvx2v"]
Oct 30 21:20:50.228455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.228425    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"]
Oct 30 21:20:50.231093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.231068    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcfca55-cbb6f5fbc-87tlm"]
Oct 30 21:20:50.258728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.258695    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"]
Oct 30 21:20:50.261300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.261271    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb245e02-fvbjx"]
Oct 30 21:20:50.868877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:50.868807    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm" podUID="ef2730fa-c259-444b-ab43-0685a70d6dc5" containerName="payload-processing" containerID="cri-o://c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981" gracePeriod=25
Oct 30 21:20:51.031944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.031910    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"]
Oct 30 21:20:51.141372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.141342    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:51.157684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.153877    6695 generic.go:356] "Generic (PLEG): container finished" podID="ef2730fa-c259-444b-ab43-0685a70d6dc5" containerID="c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981" exitCode=0
Oct 30 21:20:51.157684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.153913    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm" event={"ID":"ef2730fa-c259-444b-ab43-0685a70d6dc5","Type":"ContainerDied","Data":"c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981"}
Oct 30 21:20:51.157684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.153938    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"
Oct 30 21:20:51.157684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.153958    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm" event={"ID":"ef2730fa-c259-444b-ab43-0685a70d6dc5","Type":"ContainerDied","Data":"16a04bda16a97f9873b7189685c88ced1f2d211b1e18477516573fa87e03a1e9"}
Oct 30 21:20:51.157684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.153979    6695 scope.go:122] "RemoveContainer" containerID="c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981"
Oct 30 21:20:51.164309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.164286    6695 scope.go:122] "RemoveContainer" containerID="c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981"
Oct 30 21:20:51.164586 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:20:51.164562    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981\": container with ID starting with c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981 not found: ID does not exist" containerID="c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981"
Oct 30 21:20:51.164675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.164597    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981"} err="failed to get container status \"c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981\": rpc error: code = NotFound desc = could not find container \"c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981\": container with ID starting with c4734dd9a7141dde4514f40d1e9a07ecce769e15ab02d0d345f45afb0dae4981 not found: ID does not exist"
Oct 30 21:20:51.215734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.215693    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a328bd-8b3e-4cff-8dcd-5ac83e8d7608" path="/var/lib/kubelet/pods/12a328bd-8b3e-4cff-8dcd-5ac83e8d7608/volumes"
Oct 30 21:20:51.216248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.216228    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02759ac-23b6-4e4f-a222-a3372dcfbab8" path="/var/lib/kubelet/pods/b02759ac-23b6-4e4f-a222-a3372dcfbab8/volumes"
Oct 30 21:20:51.216718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.216701    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" path="/var/lib/kubelet/pods/ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73/volumes"
Oct 30 21:20:51.250284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.250136    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume\") pod \"ef2730fa-c259-444b-ab43-0685a70d6dc5\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") "
Oct 30 21:20:51.250284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.250235    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj\" (UniqueName: \"kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj\") pod \"ef2730fa-c259-444b-ab43-0685a70d6dc5\" (UID: \"ef2730fa-c259-444b-ab43-0685a70d6dc5\") "
Oct 30 21:20:51.255424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.251567    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume" pod "ef2730fa-c259-444b-ab43-0685a70d6dc5" (UID: "ef2730fa-c259-444b-ab43-0685a70d6dc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:51.262529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.262490    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj" pod "ef2730fa-c259-444b-ab43-0685a70d6dc5" (UID: "ef2730fa-c259-444b-ab43-0685a70d6dc5"). InnerVolumeSpecName "kube-api-access-tmpfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:51.351447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.351415    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2730fa-c259-444b-ab43-0685a70d6dc5-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:51.351447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.351450    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpfj\" (UniqueName: \"kubernetes.io/projected/ef2730fa-c259-444b-ab43-0685a70d6dc5-kube-api-access-tmpfj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:51.497716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.497677    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"]
Oct 30 21:20:51.500553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.500518    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bb245e02-f9c4fd59f-s7cdm"]
Oct 30 21:20:51.920668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.915109    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"]
Oct 30 21:20:51.922675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.922646    6695 cpu_manager.go:414] "containerMap: removing container" podUID="aa9afa61-0c3d-431c-92b3-5619081096ac" containerName="revoke-keys"
Oct 30 21:20:51.922858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.922843    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9afa61-0c3d-431c-92b3-5619081096ac" containerName="revoke-keys"
Oct 30 21:20:51.922994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.922980    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b02759ac-23b6-4e4f-a222-a3372dcfbab8" containerName="payload-processing"
Oct 30 21:20:51.923086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923074    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02759ac-23b6-4e4f-a222-a3372dcfbab8" containerName="payload-processing"
Oct 30 21:20:51.923180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923158    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" containerName="payload-pre-processing"
Oct 30 21:20:51.923180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923174    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" containerName="payload-pre-processing"
Oct 30 21:20:51.923370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923199    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ef2730fa-c259-444b-ab43-0685a70d6dc5" containerName="payload-processing"
Oct 30 21:20:51.923370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923213    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2730fa-c259-444b-ab43-0685a70d6dc5" containerName="payload-processing"
Oct 30 21:20:51.923370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923347    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02759ac-23b6-4e4f-a222-a3372dcfbab8" containerName="payload-processing"
Oct 30 21:20:51.923370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923361    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd9a80c-c1ed-4f9f-a018-e0ed43b23b73" containerName="payload-pre-processing"
Oct 30 21:20:51.923568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923374    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2730fa-c259-444b-ab43-0685a70d6dc5" containerName="payload-processing"
Oct 30 21:20:51.923568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.923386    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9afa61-0c3d-431c-92b3-5619081096ac" containerName="revoke-keys"
Oct 30 21:20:51.927045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.927021    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:51.942809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.942667    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:20:51.942809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.942704    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:20:51.946036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.945867    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"]
Oct 30 21:20:51.957758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.957731    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:20:51.957900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:51.957808    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:20:52.059083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.059047    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.059243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.059089    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5ls\" (UniqueName: \"kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.059243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.059229    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.160468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.160431    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.161082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.161047    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5ls\" (UniqueName: \"kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.161239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.161220    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.161318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.161266    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.163125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.163099    6695 generic.go:356] "Generic (PLEG): container finished" podID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerID="f48a3d939ef03fcec6c06c9b3f922d7a4d7de39e949c44e2721898a9ffe47a34" exitCode=0
Oct 30 21:20:52.163240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.163187    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" event={"ID":"4d7dae51-165f-4ba4-93b3-979c00d3e0e5","Type":"ContainerDied","Data":"f48a3d939ef03fcec6c06c9b3f922d7a4d7de39e949c44e2721898a9ffe47a34"}
Oct 30 21:20:52.163730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.163708    6695 scope.go:122] "RemoveContainer" containerID="f48a3d939ef03fcec6c06c9b3f922d7a4d7de39e949c44e2721898a9ffe47a34"
Oct 30 21:20:52.163971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.163951    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.165076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.165052    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:52.165153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.165121    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:52.165190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.165154    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:52.180528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.180506    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5ls\" (UniqueName: \"kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls\") pod \"maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.252674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.252645    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:52.517824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.517748    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"]
Oct 30 21:20:52.522150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.522119    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bdcfca55-bfgnd"]
Oct 30 21:20:52.556296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:52.556263    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"]
Oct 30 21:20:52.561040 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:52.561009    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75406097_57ea_4b8b_916e_b4a33095d22f.slice/crio-6602b471098c41c4508a91bcc757f144372c3e968888cf5aca4bb073c21090a6 WatchSource:0}: Error finding container 6602b471098c41c4508a91bcc757f144372c3e968888cf5aca4bb073c21090a6: Status 404 returned error can't find the container with id 6602b471098c41c4508a91bcc757f144372c3e968888cf5aca4bb073c21090a6
Oct 30 21:20:53.171850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.171746    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787" event={"ID":"75406097-57ea-4b8b-916e-b4a33095d22f","Type":"ContainerStarted","Data":"c6b36fb36a0da434f4a62e84a4c337fcf979f022f4d6fdec43b9f0e8488700fe"}
Oct 30 21:20:53.171850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.171812    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787" event={"ID":"75406097-57ea-4b8b-916e-b4a33095d22f","Type":"ContainerStarted","Data":"6602b471098c41c4508a91bcc757f144372c3e968888cf5aca4bb073c21090a6"}
Oct 30 21:20:53.174209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.174177    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" event={"ID":"4d7dae51-165f-4ba4-93b3-979c00d3e0e5","Type":"ContainerStarted","Data":"fa417cdfab07485afa904fdb4a0894e398e7da4b796f197f4934c6c2441a0bec"}
Oct 30 21:20:53.236630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.227976    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c9ebf3-23b1-4491-adb5-1fe9102ebb64" path="/var/lib/kubelet/pods/75c9ebf3-23b1-4491-adb5-1fe9102ebb64/volumes"
Oct 30 21:20:53.236630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.228497    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2730fa-c259-444b-ab43-0685a70d6dc5" path="/var/lib/kubelet/pods/ef2730fa-c259-444b-ab43-0685a70d6dc5/volumes"
Oct 30 21:20:53.237932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.237860    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787" podStartSLOduration=2.237837139 podStartE2EDuration="2.237837139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:53.231647848 +0000 UTC m=+636.154106147" watchObservedRunningTime="2026-10-30 21:20:53.237837139 +0000 UTC m=+636.160295439"
Oct 30 21:20:53.250651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.250588    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:53.251750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.251728    6695 patch_prober.go:29] interesting pod/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:20:53.251835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.251775    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:20:53.327627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.326787    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"]
Oct 30 21:20:53.342553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.342030    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.356939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.354785    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"]
Oct 30 21:20:53.361713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.361685    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea40ee-openshift-default-dockercfg-8mf6h\""
Oct 30 21:20:53.399827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.399791    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.399827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.399834    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.399852    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.400026    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.400068    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.400119    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.400158    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfmj\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.400177    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.400256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.400244    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.502880    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.502956    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.502996    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.503083    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfmj\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.503130    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.503181    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.503259    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.503302    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.503658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.503327    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.504919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.504889    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.505446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.505236    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.505807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.505782    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.506167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.506147    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.506304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.505796    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.508917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.508893    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.543154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.543077    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.543840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.543814    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.545500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.545475    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfmj\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:53.662995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:53.662964    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:54.036618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.036554    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"]
Oct 30 21:20:54.049534 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:20:54.049503    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244dad53_4bc6_4800_a237_25ed5be27b5d.slice/crio-8213d96a4307eccb49bd11d6349796836f6d5b71009e0cefe16488d947a48ef1 WatchSource:0}: Error finding container 8213d96a4307eccb49bd11d6349796836f6d5b71009e0cefe16488d947a48ef1: Status 404 returned error can't find the container with id 8213d96a4307eccb49bd11d6349796836f6d5b71009e0cefe16488d947a48ef1
Oct 30 21:20:54.051855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.051818    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:54.051964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.051903    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:54.051964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.051948    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:54.182448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.182404    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" event={"ID":"244dad53-4bc6-4800-a237-25ed5be27b5d","Type":"ContainerStarted","Data":"6d8bec46dcdf1c45bfe92b7566273f3f21fb15e2053c6879d67f2d45f675d6ab"}
Oct 30 21:20:54.182949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.182478    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" event={"ID":"244dad53-4bc6-4800-a237-25ed5be27b5d","Type":"ContainerStarted","Data":"8213d96a4307eccb49bd11d6349796836f6d5b71009e0cefe16488d947a48ef1"}
Oct 30 21:20:54.184134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.184110    6695 generic.go:356] "Generic (PLEG): container finished" podID="75406097-57ea-4b8b-916e-b4a33095d22f" containerID="c6b36fb36a0da434f4a62e84a4c337fcf979f022f4d6fdec43b9f0e8488700fe" exitCode=0
Oct 30 21:20:54.184539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.184514    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787" event={"ID":"75406097-57ea-4b8b-916e-b4a33095d22f","Type":"ContainerDied","Data":"c6b36fb36a0da434f4a62e84a4c337fcf979f022f4d6fdec43b9f0e8488700fe"}
Oct 30 21:20:54.239630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.232849    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podStartSLOduration=1.23282817 podStartE2EDuration="1.23282817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:54.232118674 +0000 UTC m=+637.154576973" watchObservedRunningTime="2026-10-30 21:20:54.23282817 +0000 UTC m=+637.155286473"
Oct 30 21:20:54.266673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.259010    6695 patch_prober.go:29] interesting pod/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:20:54.266673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.259080    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:20:54.669796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:54.669709    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:55.035629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.032832    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"]
Oct 30 21:20:55.056628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.052222    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b72f7bd8-wkb7p"]
Oct 30 21:20:55.243085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.243048    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf1c7c9-c4f2-410e-b867-8f763dc15f0e" path="/var/lib/kubelet/pods/6cf1c7c9-c4f2-410e-b867-8f763dc15f0e/volumes"
Oct 30 21:20:55.250471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.250439    6695 patch_prober.go:29] interesting pod/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:20:55.250654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.250495    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:20:55.398543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.398511    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:55.452429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.452395    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca\") pod \"75406097-57ea-4b8b-916e-b4a33095d22f\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") "
Oct 30 21:20:55.452641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.452504    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls\" (UniqueName: \"kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls\") pod \"75406097-57ea-4b8b-916e-b4a33095d22f\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") "
Oct 30 21:20:55.452641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.452535    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca\") pod \"75406097-57ea-4b8b-916e-b4a33095d22f\" (UID: \"75406097-57ea-4b8b-916e-b4a33095d22f\") "
Oct 30 21:20:55.452855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.452830    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca" pod "75406097-57ea-4b8b-916e-b4a33095d22f" (UID: "75406097-57ea-4b8b-916e-b4a33095d22f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:55.454749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.454714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca" pod "75406097-57ea-4b8b-916e-b4a33095d22f" (UID: "75406097-57ea-4b8b-916e-b4a33095d22f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:55.454850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.454761    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls" pod "75406097-57ea-4b8b-916e-b4a33095d22f" (UID: "75406097-57ea-4b8b-916e-b4a33095d22f"). InnerVolumeSpecName "kube-api-access-tl5ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:55.553850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.553811    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:55.553850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.553851    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5ls\" (UniqueName: \"kubernetes.io/projected/75406097-57ea-4b8b-916e-b4a33095d22f-kube-api-access-tl5ls\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:55.554098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.553864    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75406097-57ea-4b8b-916e-b4a33095d22f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:55.669828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.669748    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:55.928682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:55.927714    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:56.033628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.032680    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr" podUID="45312c58-fec3-45fe-b45d-5259a69c0007" containerName="payload-pre-processing" containerID="cri-o://64c6f4d09383b9965dbd6349d8151eb827a55c8f5f53164ff89b9ac9e20dbec9" gracePeriod=25
Oct 30 21:20:56.208139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.208042    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787" event={"ID":"75406097-57ea-4b8b-916e-b4a33095d22f","Type":"ContainerDied","Data":"6602b471098c41c4508a91bcc757f144372c3e968888cf5aca4bb073c21090a6"}
Oct 30 21:20:56.208139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.208098    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6602b471098c41c4508a91bcc757f144372c3e968888cf5aca4bb073c21090a6"
Oct 30 21:20:56.208412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.208168    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"
Oct 30 21:20:56.219733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.218183    6695 generic.go:356] "Generic (PLEG): container finished" podID="45312c58-fec3-45fe-b45d-5259a69c0007" containerID="64c6f4d09383b9965dbd6349d8151eb827a55c8f5f53164ff89b9ac9e20dbec9" exitCode=0
Oct 30 21:20:56.219733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.219436    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr" event={"ID":"45312c58-fec3-45fe-b45d-5259a69c0007","Type":"ContainerDied","Data":"64c6f4d09383b9965dbd6349d8151eb827a55c8f5f53164ff89b9ac9e20dbec9"}
Oct 30 21:20:56.219733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.219504    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:56.253959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.253189    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:20:56.253959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.253515    6695 patch_prober.go:29] interesting pod/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:20:56.253959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.253565    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:20:56.446644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.424830    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:56.468667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.468530    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume\") pod \"45312c58-fec3-45fe-b45d-5259a69c0007\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") "
Oct 30 21:20:56.468857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.468723    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc\" (UniqueName: \"kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc\") pod \"45312c58-fec3-45fe-b45d-5259a69c0007\" (UID: \"45312c58-fec3-45fe-b45d-5259a69c0007\") "
Oct 30 21:20:56.469357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.469290    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume" pod "45312c58-fec3-45fe-b45d-5259a69c0007" (UID: "45312c58-fec3-45fe-b45d-5259a69c0007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:56.471943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.471916    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc" pod "45312c58-fec3-45fe-b45d-5259a69c0007" (UID: "45312c58-fec3-45fe-b45d-5259a69c0007"). InnerVolumeSpecName "kube-api-access-86fzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:56.570499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.570458    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45312c58-fec3-45fe-b45d-5259a69c0007-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:56.570499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:56.570502    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fzc\" (UniqueName: \"kubernetes.io/projected/45312c58-fec3-45fe-b45d-5259a69c0007-kube-api-access-86fzc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.244896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.244861    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr" event={"ID":"45312c58-fec3-45fe-b45d-5259a69c0007","Type":"ContainerDied","Data":"ebc81c4023126b8a30a903cae9e00a28d9cc4d24e12d00ea8476bb2554fa9a7f"}
Oct 30 21:20:57.244896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.244891    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"
Oct 30 21:20:57.245153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.244939    6695 scope.go:122] "RemoveContainer" containerID="64c6f4d09383b9965dbd6349d8151eb827a55c8f5f53164ff89b9ac9e20dbec9"
Oct 30 21:20:57.250156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.250126    6695 patch_prober.go:29] interesting pod/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:20:57.250384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.250352    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:20:57.307845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.307790    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"]
Oct 30 21:20:57.308452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.308418    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b72f7bd8-8d5dc4dff-bmjfr"]
Oct 30 21:20:57.809274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.809221    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"]
Oct 30 21:20:57.834631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.824350    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-acb99762-46xpw"]
Oct 30 21:20:57.964717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.964672    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"]
Oct 30 21:20:57.965039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:57.965005    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy" containerID="cri-o://23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc" gracePeriod=30
Oct 30 21:20:58.250404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:58.250318    6695 patch_prober.go:29] interesting pod/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:20:58.250404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:58.250374    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 21:20:59.213684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:59.213566    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45312c58-fec3-45fe-b45d-5259a69c0007" path="/var/lib/kubelet/pods/45312c58-fec3-45fe-b45d-5259a69c0007/volumes"
Oct 30 21:20:59.214067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:59.214047    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9afa61-0c3d-431c-92b3-5619081096ac" path="/var/lib/kubelet/pods/aa9afa61-0c3d-431c-92b3-5619081096ac/volumes"
Oct 30 21:20:59.255596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:59.255562    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:20:59.256745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:20:59.256721    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:21:01.080506 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:21:01.080464    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:21:02.369168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:02.369127    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"]
Oct 30 21:21:02.406304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:02.406260    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"]
Oct 30 21:21:03.247918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.247842    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:21:03.284384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.284348    6695 generic.go:356] "Generic (PLEG): container finished" podID="741905db-4c68-4b87-ab48-a3c708449e40" containerID="23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc" exitCode=0
Oct 30 21:21:03.284575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.284397    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"
Oct 30 21:21:03.284575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.284410    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" event={"ID":"741905db-4c68-4b87-ab48-a3c708449e40","Type":"ContainerDied","Data":"23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc"}
Oct 30 21:21:03.284575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.284459    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw" event={"ID":"741905db-4c68-4b87-ab48-a3c708449e40","Type":"ContainerDied","Data":"2712babc6697480d3cdc40bf05b7b2b1d0815cd0c34960afcad5c75a116f74f6"}
Oct 30 21:21:03.284575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.284482    6695 scope.go:122] "RemoveContainer" containerID="23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc"
Oct 30 21:21:03.293874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.293848    6695 scope.go:122] "RemoveContainer" containerID="2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"
Oct 30 21:21:03.303777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.303748    6695 scope.go:122] "RemoveContainer" containerID="23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc"
Oct 30 21:21:03.304179 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:21:03.304148    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc\": container with ID starting with 23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc not found: ID does not exist" containerID="23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc"
Oct 30 21:21:03.304271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.304194    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc"} err="failed to get container status \"23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc\": rpc error: code = NotFound desc = could not find container \"23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc\": container with ID starting with 23c6edcd71ed153618d8e2a9f0e557e78c6e827a841ab2f1e51b484d958b53bc not found: ID does not exist"
Oct 30 21:21:03.304271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.304221    6695 scope.go:122] "RemoveContainer" containerID="2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"
Oct 30 21:21:03.305016 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:21:03.304590    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100\": container with ID starting with 2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100 not found: ID does not exist" containerID="2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"
Oct 30 21:21:03.305159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.305030    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100"} err="failed to get container status \"2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100\": rpc error: code = NotFound desc = could not find container \"2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100\": container with ID starting with 2278586bc356c89faf5f11429bed8223c4e4a4b84dc81da8766bfdffc7afc100 not found: ID does not exist"
Oct 30 21:21:03.359002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.358966    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359017    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359054    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359072    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359096    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359137    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359225    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359273    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359318    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:03.359545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359411    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo\") pod \"741905db-4c68-4b87-ab48-a3c708449e40\" (UID: \"741905db-4c68-4b87-ab48-a3c708449e40\") "
Oct 30 21:21:03.359545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359425    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:03.359545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359526    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:03.359863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359672    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:03.359921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359864    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:03.359921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359874    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.359921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359895    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.359921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359908    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.360114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.359922    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.361573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.361543    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:03.361573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.361556    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:03.361769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.361742    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "kube-api-access-jmgzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:03.361948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.361934    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo" pod "741905db-4c68-4b87-ab48-a3c708449e40" (UID: "741905db-4c68-4b87-ab48-a3c708449e40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:03.460504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.460462    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741905db-4c68-4b87-ab48-a3c708449e40-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.460504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.460496    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.460504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.460506    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmgzp\" (UniqueName: \"kubernetes.io/projected/741905db-4c68-4b87-ab48-a3c708449e40-kube-api-access-jmgzp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.460504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.460515    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741905db-4c68-4b87-ab48-a3c708449e40-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.461022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.460523    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741905db-4c68-4b87-ab48-a3c708449e40-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:03.647669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.647616    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"]
Oct 30 21:21:03.658019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.657983    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-acb99762-openshift-default-6ddfb8db5f-ps6tw"]
Oct 30 21:21:03.894958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.894871    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"]
Oct 30 21:21:03.895549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895526    6695 cpu_manager.go:414] "containerMap: removing container" podUID="45312c58-fec3-45fe-b45d-5259a69c0007" containerName="payload-pre-processing"
Oct 30 21:21:03.895549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895548    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45312c58-fec3-45fe-b45d-5259a69c0007" containerName="payload-pre-processing"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895560    6695 cpu_manager.go:414] "containerMap: removing container" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895566    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895583    6695 cpu_manager.go:414] "containerMap: removing container" podUID="75406097-57ea-4b8b-916e-b4a33095d22f" containerName="revoke-keys"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895587    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75406097-57ea-4b8b-916e-b4a33095d22f" containerName="revoke-keys"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895598    6695 cpu_manager.go:414] "containerMap: removing container" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895634    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895706    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="45312c58-fec3-45fe-b45d-5259a69c0007" containerName="payload-pre-processing"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895713    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy"
Oct 30 21:21:03.895729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.895720    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="75406097-57ea-4b8b-916e-b4a33095d22f" containerName="revoke-keys"
Oct 30 21:21:03.899149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.899128    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:03.917283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.917258    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-49738b\""
Oct 30 21:21:03.917639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.917592    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-49738b-dockercfg-4jnm4\""
Oct 30 21:21:03.920061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.920033    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"]
Oct 30 21:21:03.993627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.969210    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:03.993627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.969342    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhkl\" (UniqueName: \"kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl\") pod \"payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:03.993627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.976856    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"]
Oct 30 21:21:03.993627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.977874    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="741905db-4c68-4b87-ab48-a3c708449e40" containerName="istio-proxy"
Oct 30 21:21:03.993627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.981437    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:03.997628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:03.995105    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"]
Oct 30 21:21:04.095176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.095133    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:04.095390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.095185    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx8m\" (UniqueName: \"kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m\") pod \"payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.095390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.095249    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhkl\" (UniqueName: \"kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl\") pod \"payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:04.095390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.095325    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.095902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.095883    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:04.114226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.114192    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhkl\" (UniqueName: \"kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl\") pod \"payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:04.196318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.196212    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.196511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.196382    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx8m\" (UniqueName: \"kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m\") pod \"payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.196990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.196969    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.212223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.212188    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:04.215931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.215902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx8m\" (UniqueName: \"kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m\") pod \"payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.332986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.332958    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:04.517950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.517916    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"]
Oct 30 21:21:04.635079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:04.635020    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"]
Oct 30 21:21:04.639157 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:04.639128    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6857618f_f853_4559_83a3_8720a280c353.slice/crio-ed829e61f514e2821da1b1eee83ba20009af9a4952b85c1768d7edf39f76b494 WatchSource:0}: Error finding container ed829e61f514e2821da1b1eee83ba20009af9a4952b85c1768d7edf39f76b494: Status 404 returned error can't find the container with id ed829e61f514e2821da1b1eee83ba20009af9a4952b85c1768d7edf39f76b494
Oct 30 21:21:05.212330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.212300    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741905db-4c68-4b87-ab48-a3c708449e40" path="/var/lib/kubelet/pods/741905db-4c68-4b87-ab48-a3c708449e40/volumes"
Oct 30 21:21:05.297716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.297675    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" event={"ID":"97034497-4f1e-4a41-9030-b2f4ab146e9f","Type":"ContainerStarted","Data":"919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261"}
Oct 30 21:21:05.297918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.297739    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" event={"ID":"97034497-4f1e-4a41-9030-b2f4ab146e9f","Type":"ContainerStarted","Data":"4934a99a70e4caf93d408fbc1cc3d612347608945f976a2061a766463b8fa27f"}
Oct 30 21:21:05.297918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.297864    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:05.299423 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.299397    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" event={"ID":"6857618f-f853-4559-83a3-8720a280c353","Type":"ContainerStarted","Data":"dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551"}
Oct 30 21:21:05.299549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.299433    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" event={"ID":"6857618f-f853-4559-83a3-8720a280c353","Type":"ContainerStarted","Data":"ed829e61f514e2821da1b1eee83ba20009af9a4952b85c1768d7edf39f76b494"}
Oct 30 21:21:05.299549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.299537    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:05.338931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.338873    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" podStartSLOduration=2.338856056 podStartE2EDuration="2.338856056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:05.338269861 +0000 UTC m=+648.260728161" watchObservedRunningTime="2026-10-30 21:21:05.338856056 +0000 UTC m=+648.261314353"
Oct 30 21:21:05.376040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:05.375976    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" podStartSLOduration=2.375960384 podStartE2EDuration="2.375960384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:05.374689493 +0000 UTC m=+648.297147822" watchObservedRunningTime="2026-10-30 21:21:05.375960384 +0000 UTC m=+648.298418686"
Oct 30 21:21:07.094987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.094949    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"]
Oct 30 21:21:07.100507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.100472    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.120458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.118066    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"]
Oct 30 21:21:07.125164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.125129    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9d677d-dockercfg-4r49b\""
Oct 30 21:21:07.125342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.125190    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9d677d\""
Oct 30 21:21:07.248626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.246313    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.248626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.246434    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfjl\" (UniqueName: \"kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl\") pod \"payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.348246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.348138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.348416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.348247    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfjl\" (UniqueName: \"kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl\") pod \"payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.349649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.349617    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.369860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.369780    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp" podUID="4d31f1d6-db41-4edc-a453-2de2efc6aa26" containerName="payload-processing" containerID="cri-o://033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6" gracePeriod=25
Oct 30 21:21:07.372701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.372289    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfjl\" (UniqueName: \"kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl\") pod \"payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.407051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.407002    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr" podUID="43b5ef49-9cda-491f-9eb5-a0f1b333717e" containerName="payload-pre-processing" containerID="cri-o://01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e" gracePeriod=25
Oct 30 21:21:07.432167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.431842    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:07.788182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.788142    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"]
Oct 30 21:21:07.951889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:07.951764    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:21:08.000260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.000235    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:21:08.005813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.005787    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume\") pod \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") "
Oct 30 21:21:08.005962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.005851    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf\" (UniqueName: \"kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf\") pod \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\" (UID: \"4d31f1d6-db41-4edc-a453-2de2efc6aa26\") "
Oct 30 21:21:08.019625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.006614    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume" pod "4d31f1d6-db41-4edc-a453-2de2efc6aa26" (UID: "4d31f1d6-db41-4edc-a453-2de2efc6aa26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:08.019625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.006835    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d31f1d6-db41-4edc-a453-2de2efc6aa26-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:08.019625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.009165    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf" pod "4d31f1d6-db41-4edc-a453-2de2efc6aa26" (UID: "4d31f1d6-db41-4edc-a453-2de2efc6aa26"). InnerVolumeSpecName "kube-api-access-n7jgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:08.084490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.084415    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"]
Oct 30 21:21:08.086855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.086828    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0a3ffe1c-8l787"]
Oct 30 21:21:08.108145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.108110    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume\") pod \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") "
Oct 30 21:21:08.108648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.108194    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr\") pod \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\" (UID: \"43b5ef49-9cda-491f-9eb5-a0f1b333717e\") "
Oct 30 21:21:08.108648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.108392    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jgf\" (UniqueName: \"kubernetes.io/projected/4d31f1d6-db41-4edc-a453-2de2efc6aa26-kube-api-access-n7jgf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:08.108648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.108587    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume" pod "43b5ef49-9cda-491f-9eb5-a0f1b333717e" (UID: "43b5ef49-9cda-491f-9eb5-a0f1b333717e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:08.110288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.110265    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr" pod "43b5ef49-9cda-491f-9eb5-a0f1b333717e" (UID: "43b5ef49-9cda-491f-9eb5-a0f1b333717e"). InnerVolumeSpecName "kube-api-access-h5qmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:08.209184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.209102    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/43b5ef49-9cda-491f-9eb5-a0f1b333717e-kube-api-access-h5qmr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:08.209370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.209350    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5ef49-9cda-491f-9eb5-a0f1b333717e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:08.242099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.237645    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"]
Oct 30 21:21:08.242099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.237945    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy" containerID="cri-o://fa417cdfab07485afa904fdb4a0894e398e7da4b796f197f4934c6c2441a0bec" gracePeriod=30
Oct 30 21:21:08.398259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.398217    6695 generic.go:356] "Generic (PLEG): container finished" podID="43b5ef49-9cda-491f-9eb5-a0f1b333717e" containerID="01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e" exitCode=0
Oct 30 21:21:08.398724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.398697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr" event={"ID":"43b5ef49-9cda-491f-9eb5-a0f1b333717e","Type":"ContainerDied","Data":"01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e"}
Oct 30 21:21:08.398897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.398882    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr" event={"ID":"43b5ef49-9cda-491f-9eb5-a0f1b333717e","Type":"ContainerDied","Data":"41482e5a63d524da15d4a5d3239103bb64692ebdfee329073e1476e0adb2ee78"}
Oct 30 21:21:08.399022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.399010    6695 scope.go:122] "RemoveContainer" containerID="01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e"
Oct 30 21:21:08.399350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.399332    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"
Oct 30 21:21:08.403991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.403965    6695 generic.go:356] "Generic (PLEG): container finished" podID="4d31f1d6-db41-4edc-a453-2de2efc6aa26" containerID="033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6" exitCode=0
Oct 30 21:21:08.404184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.404105    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"
Oct 30 21:21:08.404184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.404111    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp" event={"ID":"4d31f1d6-db41-4edc-a453-2de2efc6aa26","Type":"ContainerDied","Data":"033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6"}
Oct 30 21:21:08.404184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.404163    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp" event={"ID":"4d31f1d6-db41-4edc-a453-2de2efc6aa26","Type":"ContainerDied","Data":"ccd1acee0d83eaa99ec536b3d354323eb295eff310925d42b165e41723d4cb2c"}
Oct 30 21:21:08.409064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.408827    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" event={"ID":"3f4f8cde-572b-4d6d-9977-1713faff4872","Type":"ContainerStarted","Data":"b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f"}
Oct 30 21:21:08.409064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.408910    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" event={"ID":"3f4f8cde-572b-4d6d-9977-1713faff4872","Type":"ContainerStarted","Data":"07387c78673aaf5355d53651ac7b9d9481cc95f58e386afec48bae7ed7699d29"}
Oct 30 21:21:08.409064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.408946    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:08.442189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.442161    6695 scope.go:122] "RemoveContainer" containerID="01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e"
Oct 30 21:21:08.444863 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:21:08.443106    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e\": container with ID starting with 01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e not found: ID does not exist" containerID="01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e"
Oct 30 21:21:08.444863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.443162    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e"} err="failed to get container status \"01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e\": rpc error: code = NotFound desc = could not find container \"01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e\": container with ID starting with 01a58a8bab6c55600c90f5d67722f2fb6e400d56d2dca4c6c1812b576747944e not found: ID does not exist"
Oct 30 21:21:08.444863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.443187    6695 scope.go:122] "RemoveContainer" containerID="033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6"
Oct 30 21:21:08.453903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.453883    6695 scope.go:122] "RemoveContainer" containerID="033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6"
Oct 30 21:21:08.454202 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:21:08.454165    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6\": container with ID starting with 033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6 not found: ID does not exist" containerID="033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6"
Oct 30 21:21:08.454246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.454216    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6"} err="failed to get container status \"033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6\": rpc error: code = NotFound desc = could not find container \"033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6\": container with ID starting with 033b7d500a0b1d5b050cecb2740295c243933d6e728c4b659b403610375297f6 not found: ID does not exist"
Oct 30 21:21:08.493492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.493377    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" podStartSLOduration=1.49336351 podStartE2EDuration="1.49336351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:08.492685028 +0000 UTC m=+651.415143337" watchObservedRunningTime="2026-10-30 21:21:08.49336351 +0000 UTC m=+651.415821807"
Oct 30 21:21:08.537191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.537114    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"]
Oct 30 21:21:08.539262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.539238    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0a3ffe1c-7dd98fbd9-jbzwp"]
Oct 30 21:21:08.567179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.567142    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"]
Oct 30 21:21:08.571849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:08.571820    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0a3ffe1c-978bbfdf5-4dglr"]
Oct 30 21:21:09.237974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.237930    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b5ef49-9cda-491f-9eb5-a0f1b333717e" path="/var/lib/kubelet/pods/43b5ef49-9cda-491f-9eb5-a0f1b333717e/volumes"
Oct 30 21:21:09.238532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.238511    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d31f1d6-db41-4edc-a453-2de2efc6aa26" path="/var/lib/kubelet/pods/4d31f1d6-db41-4edc-a453-2de2efc6aa26/volumes"
Oct 30 21:21:09.238970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.238950    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75406097-57ea-4b8b-916e-b4a33095d22f" path="/var/lib/kubelet/pods/75406097-57ea-4b8b-916e-b4a33095d22f/volumes"
Oct 30 21:21:09.461080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461044    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"]
Oct 30 21:21:09.461738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461716    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4d31f1d6-db41-4edc-a453-2de2efc6aa26" containerName="payload-processing"
Oct 30 21:21:09.461801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461741    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d31f1d6-db41-4edc-a453-2de2efc6aa26" containerName="payload-processing"
Oct 30 21:21:09.461801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461789    6695 cpu_manager.go:414] "containerMap: removing container" podUID="43b5ef49-9cda-491f-9eb5-a0f1b333717e" containerName="payload-pre-processing"
Oct 30 21:21:09.461801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461798    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b5ef49-9cda-491f-9eb5-a0f1b333717e" containerName="payload-pre-processing"
Oct 30 21:21:09.461917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461907    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d31f1d6-db41-4edc-a453-2de2efc6aa26" containerName="payload-processing"
Oct 30 21:21:09.461949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.461926    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b5ef49-9cda-491f-9eb5-a0f1b333717e" containerName="payload-pre-processing"
Oct 30 21:21:09.467288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.467257    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.484142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.484114    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b31518-openshift-default-dockercfg-xdgb8\""
Oct 30 21:21:09.498071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.497983    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"]
Oct 30 21:21:09.560422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560385    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560423    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560452    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560640    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560681    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hc22\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560753    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560813    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560837    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.560977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.560965    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662335    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662391    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hc22\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662427    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662514    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662560    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662653    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662668    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662696    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.662900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662856    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.663241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.662985    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.663454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.663410    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.664073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.664043    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.664295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.664273    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.665957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.665936    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.666069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.665941    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.686984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.686957    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hc22\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.687483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.687463    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:09.789761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:09.783811    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:10.144319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.144157    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"]
Oct 30 21:21:10.157631 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:10.157584    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb51624_01d8_48ed_9911_99ea9d0d8207.slice/crio-b2a24ce8bab90b8106f81945c19d89b382f8e5aa645b93d0084771bbf069ad85 WatchSource:0}: Error finding container b2a24ce8bab90b8106f81945c19d89b382f8e5aa645b93d0084771bbf069ad85: Status 404 returned error can't find the container with id b2a24ce8bab90b8106f81945c19d89b382f8e5aa645b93d0084771bbf069ad85
Oct 30 21:21:10.160440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.160409    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:10.160503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.160478    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:10.160534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.160508    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:10.448466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.448311    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" event={"ID":"bbb51624-01d8-48ed-9911-99ea9d0d8207","Type":"ContainerStarted","Data":"b19fbf4b4a54a4cee257a703cca32fd3ef790f6d5270ad5e1799c9219dfda6c0"}
Oct 30 21:21:10.448466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.448453    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" event={"ID":"bbb51624-01d8-48ed-9911-99ea9d0d8207","Type":"ContainerStarted","Data":"b2a24ce8bab90b8106f81945c19d89b382f8e5aa645b93d0084771bbf069ad85"}
Oct 30 21:21:10.492949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.492882    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podStartSLOduration=1.492861228 podStartE2EDuration="1.492861228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:10.489290085 +0000 UTC m=+653.411748384" watchObservedRunningTime="2026-10-30 21:21:10.492861228 +0000 UTC m=+653.415319527"
Oct 30 21:21:10.784513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:10.784480    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:11.789961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:11.789927    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:12.461203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:12.461175    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:12.462663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:12.462634    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:13.469231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.469197    6695 generic.go:356] "Generic (PLEG): container finished" podID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerID="fa417cdfab07485afa904fdb4a0894e398e7da4b796f197f4934c6c2441a0bec" exitCode=0
Oct 30 21:21:13.469717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.469277    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" event={"ID":"4d7dae51-165f-4ba4-93b3-979c00d3e0e5","Type":"ContainerDied","Data":"fa417cdfab07485afa904fdb4a0894e398e7da4b796f197f4934c6c2441a0bec"}
Oct 30 21:21:13.469717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.469350    6695 scope.go:122] "RemoveContainer" containerID="f48a3d939ef03fcec6c06c9b3f922d7a4d7de39e949c44e2721898a9ffe47a34"
Oct 30 21:21:13.513903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.513834    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:21:13.628291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628246    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628333    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628401    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628494    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628530    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628592    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628755    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.628811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628794    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.629119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.628879    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo\") pod \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\" (UID: \"4d7dae51-165f-4ba4-93b3-979c00d3e0e5\") "
Oct 30 21:21:13.629119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.629038    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:13.629899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.629850    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:13.630126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.630084    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:13.630386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.630223    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:13.630782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.630633    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.630782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.630662    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.630782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.630677    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.630782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.630691    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.631405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.631377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:13.646132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.636768    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:13.646132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.644629    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:13.646132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.644790    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "kube-api-access-pmdq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:13.647690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.647659    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo" pod "4d7dae51-165f-4ba4-93b3-979c00d3e0e5" (UID: "4d7dae51-165f-4ba4-93b3-979c00d3e0e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:13.732031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.731994    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.732031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.732028    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.732031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.732038    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.732309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.732048    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdq4\" (UniqueName: \"kubernetes.io/projected/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-kube-api-access-pmdq4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:13.732309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:13.732058    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7dae51-165f-4ba4-93b3-979c00d3e0e5-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:21:14.476815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:14.476764    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"
Oct 30 21:21:14.477262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:14.476820    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f" event={"ID":"4d7dae51-165f-4ba4-93b3-979c00d3e0e5","Type":"ContainerDied","Data":"3c039767c9be7aca62ecb80fe9f5e778b251b3b2695952b1e89d9a3ffb9af525"}
Oct 30 21:21:14.477262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:14.476904    6695 scope.go:122] "RemoveContainer" containerID="fa417cdfab07485afa904fdb4a0894e398e7da4b796f197f4934c6c2441a0bec"
Oct 30 21:21:14.546528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:14.546487    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"]
Oct 30 21:21:14.550470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:14.550441    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0a3ffe1c-openshift-default-56c66944b9-hcp5f"]
Oct 30 21:21:15.214230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.214191    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" path="/var/lib/kubelet/pods/4d7dae51-165f-4ba4-93b3-979c00d3e0e5/volumes"
Oct 30 21:21:15.434090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.434050    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"]
Oct 30 21:21:15.434763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.434743    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy"
Oct 30 21:21:15.434763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.434762    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy"
Oct 30 21:21:15.434925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.434858    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy"
Oct 30 21:21:15.439620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.439583    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.449055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.449029    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmwx\" (UniqueName: \"kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx\") pod \"payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.449171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.449088    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.455009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.454978    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"]
Oct 30 21:21:15.463964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.463932    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b31518\""
Oct 30 21:21:15.495709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.495203    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"]
Oct 30 21:21:15.496754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.496729    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy"
Oct 30 21:21:15.496754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.496753    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy"
Oct 30 21:21:15.496888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.496865    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7dae51-165f-4ba4-93b3-979c00d3e0e5" containerName="istio-proxy"
Oct 30 21:21:15.500077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.500057    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.522176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.522142    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"]
Oct 30 21:21:15.538618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.538581    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b31518-dockercfg-kgn9f\""
Oct 30 21:21:15.549865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.549833    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhg8s\" (UniqueName: \"kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s\") pod \"payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.550037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.549902    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmwx\" (UniqueName: \"kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx\") pod \"payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.550037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.550019    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.550163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.550077    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.550712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.550694    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.570483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.570445    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmwx\" (UniqueName: \"kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx\") pod \"payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.651466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.651404    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhg8s\" (UniqueName: \"kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s\") pod \"payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.651718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.651565    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.652342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.652320    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.672303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.672268    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhg8s\" (UniqueName: \"kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s\") pod \"payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:15.755928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.755772    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:15.815313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:15.815271    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:16.077582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.077540    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"]
Oct 30 21:21:16.082192 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:16.082163    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4bc7cf_96c4_4da2_ae1e_e4e99e554643.slice/crio-03e35a6ced9a6519a1374f1059c52baaffc939956c23e8d3efe8e745cdb733d7 WatchSource:0}: Error finding container 03e35a6ced9a6519a1374f1059c52baaffc939956c23e8d3efe8e745cdb733d7: Status 404 returned error can't find the container with id 03e35a6ced9a6519a1374f1059c52baaffc939956c23e8d3efe8e745cdb733d7
Oct 30 21:21:16.149851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.149818    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"]
Oct 30 21:21:16.160467 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:16.154790    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960d0dfc_ea23_4db4_9a2c_12b459af443e.slice/crio-acea1d8d0e13c2d1fc72a1d28b1d34b9d6ef4f1e9014afceade09bbc90c69859 WatchSource:0}: Error finding container acea1d8d0e13c2d1fc72a1d28b1d34b9d6ef4f1e9014afceade09bbc90c69859: Status 404 returned error can't find the container with id acea1d8d0e13c2d1fc72a1d28b1d34b9d6ef4f1e9014afceade09bbc90c69859
Oct 30 21:21:16.317874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.317845    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 21:21:16.318078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.318062    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 21:21:16.492903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.492804    6695 generic.go:356] "Generic (PLEG): container finished" podID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerID="6d8bec46dcdf1c45bfe92b7566273f3f21fb15e2053c6879d67f2d45f675d6ab" exitCode=0
Oct 30 21:21:16.492903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.492872    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" event={"ID":"244dad53-4bc6-4800-a237-25ed5be27b5d","Type":"ContainerDied","Data":"6d8bec46dcdf1c45bfe92b7566273f3f21fb15e2053c6879d67f2d45f675d6ab"}
Oct 30 21:21:16.493378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.493356    6695 scope.go:122] "RemoveContainer" containerID="6d8bec46dcdf1c45bfe92b7566273f3f21fb15e2053c6879d67f2d45f675d6ab"
Oct 30 21:21:16.494743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.494717    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" event={"ID":"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643","Type":"ContainerStarted","Data":"e2e68297cbc3556a5e294773124501153ff096c21bf67f90b5ab0f3400693b03"}
Oct 30 21:21:16.494868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.494771    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" event={"ID":"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643","Type":"ContainerStarted","Data":"03e35a6ced9a6519a1374f1059c52baaffc939956c23e8d3efe8e745cdb733d7"}
Oct 30 21:21:16.494868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.494802    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:16.494944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.494855    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:16.494944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.494918    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:16.495009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.494958    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:16.496773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.496750    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" event={"ID":"960d0dfc-ea23-4db4-9a2c-12b459af443e","Type":"ContainerStarted","Data":"0de39b3c68e20a4c098a17370f36b86bac13833719ca71ed98e2275409d5dd41"}
Oct 30 21:21:16.497128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.496805    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" event={"ID":"960d0dfc-ea23-4db4-9a2c-12b459af443e","Type":"ContainerStarted","Data":"acea1d8d0e13c2d1fc72a1d28b1d34b9d6ef4f1e9014afceade09bbc90c69859"}
Oct 30 21:21:16.497128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.496928    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:16.589491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.587548    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" podStartSLOduration=1.587527855 podStartE2EDuration="1.587527855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:16.584899011 +0000 UTC m=+659.507357310" watchObservedRunningTime="2026-10-30 21:21:16.587527855 +0000 UTC m=+659.509986154"
Oct 30 21:21:16.621743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:16.621695    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" podStartSLOduration=1.62167894 podStartE2EDuration="1.62167894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:16.621026363 +0000 UTC m=+659.543484661" watchObservedRunningTime="2026-10-30 21:21:16.62167894 +0000 UTC m=+659.544137238"
Oct 30 21:21:17.503670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:17.503632    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" event={"ID":"244dad53-4bc6-4800-a237-25ed5be27b5d","Type":"ContainerStarted","Data":"ba45d4e95e06a74ae0b078314d8034881c342c355f13af664e58f1a27aee1b9c"}
Oct 30 21:21:17.663431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:17.663343    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:21:17.663652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:17.663590    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:21:17.663707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:17.663678    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 21:21:18.663474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:18.663439    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:21:18.663935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:18.663489    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 21:21:19.444679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:19.444649    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 21:21:19.663426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:19.663388    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:21:19.663624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:19.663459    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 21:21:20.663753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:20.663715    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:21:20.664153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:20.663774    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 21:21:21.664434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:21.664396    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:21:21.664970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:21.664470    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 21:21:22.795413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:22.795370    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:21:22.795926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:22.795857    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:21:22.797044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:22.797024    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:21:27.506511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:27.506480    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 21:21:27.507093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:27.506857    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 21:21:31.073192 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:21:31.073146    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:21:32.029772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.029732    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"]
Oct 30 21:21:32.034066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.034040    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.056332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.056297    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"]
Oct 30 21:21:32.062714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.062591    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"kube-root-ca.crt\""
Oct 30 21:21:32.062714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.062596    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"openshift-service-ca.crt\""
Oct 30 21:21:32.064042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.064017    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"default-dockercfg-8qqf5\""
Oct 30 21:21:32.066919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.066899    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"simulator-w0-49738b-kserve-self-signed-certs\""
Oct 30 21:21:32.081558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.081530    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxjw\" (UniqueName: \"kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.081987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.081567    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.081987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.081599    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.081987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.081748    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.081987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.081893    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.081987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.081929    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.183400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183358    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.183583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183424    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.183583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183500    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxjw\" (UniqueName: \"kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.183583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183525    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.183583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183543    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.183583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183578    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.184109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.183982    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.184109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.184047    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.184266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.184126    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.185782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.185759    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.186111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.186096    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.202479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.202441    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxjw\" (UniqueName: \"kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw\") pod \"simulator-w0-49738b-kserve-96d44c498-nd49z\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") " pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.354409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.354304    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:32.660822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:32.660716    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"]
Oct 30 21:21:32.665504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:32.665470    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43fcd1a9_8a47_47c9_8a06_870c1dec14b6.slice/crio-8b885d6e31cd74980ae8bc4130c230d7110d80f3980a064568ce1761207ccb3b WatchSource:0}: Error finding container 8b885d6e31cd74980ae8bc4130c230d7110d80f3980a064568ce1761207ccb3b: Status 404 returned error can't find the container with id 8b885d6e31cd74980ae8bc4130c230d7110d80f3980a064568ce1761207ccb3b
Oct 30 21:21:33.630644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:33.630583    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" event={"ID":"43fcd1a9-8a47-47c9-8a06-870c1dec14b6","Type":"ContainerStarted","Data":"e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b"}
Oct 30 21:21:33.631083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:33.630653    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" event={"ID":"43fcd1a9-8a47-47c9-8a06-870c1dec14b6","Type":"ContainerStarted","Data":"8b885d6e31cd74980ae8bc4130c230d7110d80f3980a064568ce1761207ccb3b"}
Oct 30 21:21:33.667485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:33.667427    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podStartSLOduration=1.667411567 podStartE2EDuration="1.667411567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:33.666365554 +0000 UTC m=+676.588823853" watchObservedRunningTime="2026-10-30 21:21:33.667411567 +0000 UTC m=+676.589869864"
Oct 30 21:21:34.355095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:34.355057    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:35.648359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:35.648314    6695 generic.go:356] "Generic (PLEG): container finished" podID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerID="b19fbf4b4a54a4cee257a703cca32fd3ef790f6d5270ad5e1799c9219dfda6c0" exitCode=0
Oct 30 21:21:35.649027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:35.648708    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" event={"ID":"bbb51624-01d8-48ed-9911-99ea9d0d8207","Type":"ContainerDied","Data":"b19fbf4b4a54a4cee257a703cca32fd3ef790f6d5270ad5e1799c9219dfda6c0"}
Oct 30 21:21:35.649143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:35.649124    6695 scope.go:122] "RemoveContainer" containerID="b19fbf4b4a54a4cee257a703cca32fd3ef790f6d5270ad5e1799c9219dfda6c0"
Oct 30 21:21:35.650675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:35.650645    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:35.650816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:35.650735    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:35.650816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:35.650786    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:36.656070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:36.656032    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" event={"ID":"bbb51624-01d8-48ed-9911-99ea9d0d8207","Type":"ContainerStarted","Data":"43c96c7324a062394d32df659325f98e51329fdc5b446bb86aef29bf410d9989"}
Oct 30 21:21:36.784856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:36.784820    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:36.785050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:36.785028    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:36.785095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:36.785077    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 21:21:37.784280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:37.784241    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:37.784727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:37.784298    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 21:21:38.784878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:38.784836    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:38.785259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:38.784903    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 21:21:39.784755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:39.784718    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:39.784988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:39.784783    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 21:21:40.328629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.323758    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"]
Oct 30 21:21:40.343481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.343447    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.345152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.344910    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"]
Oct 30 21:21:40.370253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.370226    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"kube-root-ca.crt\""
Oct 30 21:21:40.370404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.370249    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"simulator-w2-9d677d-kserve-self-signed-certs\""
Oct 30 21:21:40.370404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.370262    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"default-dockercfg-dxblp\""
Oct 30 21:21:40.370750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.370734    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"openshift-service-ca.crt\""
Oct 30 21:21:40.392738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.392711    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.392900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.392808    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.392952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.392925    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.392991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.392970    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9c6\" (UniqueName: \"kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.393097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.393082    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.393137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.393116    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.494506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.494466    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.494734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.494529    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.494734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.494581    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9c6\" (UniqueName: \"kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.494734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.494642    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.494734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.494679    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.494734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.494704    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.495057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.495027    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.495175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.495032    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.495175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.495105    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.496944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.496917    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.497090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.497074    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.520666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.520633    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9c6\" (UniqueName: \"kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6\") pod \"simulator-w2-9d677d-kserve-c7d7885d-nhd9k\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") " pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.658790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.658702    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:40.785062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.785029    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:40.785434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:40.785098    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 21:21:41.005840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:41.005762    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"]
Oct 30 21:21:41.009692 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:41.009664    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98b7b70a_6cbf_43fb_94ec_7108fa483027.slice/crio-449fd0b0572b4876e52c753183790793b4d53f54cf8f07516a2f6813f3428153 WatchSource:0}: Error finding container 449fd0b0572b4876e52c753183790793b4d53f54cf8f07516a2f6813f3428153: Status 404 returned error can't find the container with id 449fd0b0572b4876e52c753183790793b4d53f54cf8f07516a2f6813f3428153
Oct 30 21:21:41.690418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:41.690383    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" event={"ID":"98b7b70a-6cbf-43fb-94ec-7108fa483027","Type":"ContainerStarted","Data":"a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78"}
Oct 30 21:21:41.690418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:41.690425    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" event={"ID":"98b7b70a-6cbf-43fb-94ec-7108fa483027","Type":"ContainerStarted","Data":"449fd0b0572b4876e52c753183790793b4d53f54cf8f07516a2f6813f3428153"}
Oct 30 21:21:41.733357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:41.733286    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podStartSLOduration=1.733263242 podStartE2EDuration="1.733263242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:41.732086186 +0000 UTC m=+684.654544485" watchObservedRunningTime="2026-10-30 21:21:41.733263242 +0000 UTC m=+684.655721541"
Oct 30 21:21:41.784587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:41.784545    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:41.784790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:41.784645    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 21:21:42.355390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.355352    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:42.367979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.367955    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:42.461704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.461664    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:42.658950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.658850    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:42.707284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.707258    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:21:42.788702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.788672    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:42.789716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:42.789696    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 21:21:50.659755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:50.659715    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:50.677754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:50.677727    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:50.749098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:50.749062    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:21:53.779329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.779293    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"]
Oct 30 21:21:53.784393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.784375    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.816178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.816141    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"default-dockercfg-vrsxp\""
Oct 30 21:21:53.819907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.816892    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"premium-w4-ea40ee-kserve-self-signed-certs\""
Oct 30 21:21:53.820116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.819098    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"]
Oct 30 21:21:53.820193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.819221    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"openshift-service-ca.crt\""
Oct 30 21:21:53.820193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.819321    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"kube-root-ca.crt\""
Oct 30 21:21:53.852795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.852760    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.852960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.852855    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzrh\" (UniqueName: \"kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.853010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.852964    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.853046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.853028    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.853080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.853049    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.853168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.853150    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954319    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954395    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954418    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954454    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954491    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954643    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzrh\" (UniqueName: \"kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954800    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954847    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.954910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.954890    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.956717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.956696    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.956963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.956947    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:53.973650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:53.973626    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzrh\" (UniqueName: \"kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh\") pod \"premium-w4-ea40ee-kserve-7566878f98-ghjqv\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") " pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:54.116649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:54.116542    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:21:54.417458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:54.417371    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"]
Oct 30 21:21:54.421401 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:21:54.421371    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a7348fd_346f_4b77_a11d_2969029c5cf4.slice/crio-89e8ddfe1a3a8081bfe7bd8d3e89d855cf45daf21147b64b5f8e33b22d63e282 WatchSource:0}: Error finding container 89e8ddfe1a3a8081bfe7bd8d3e89d855cf45daf21147b64b5f8e33b22d63e282: Status 404 returned error can't find the container with id 89e8ddfe1a3a8081bfe7bd8d3e89d855cf45daf21147b64b5f8e33b22d63e282
Oct 30 21:21:54.761280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:54.761182    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" event={"ID":"5a7348fd-346f-4b77-a11d-2969029c5cf4","Type":"ContainerStarted","Data":"e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659"}
Oct 30 21:21:54.761280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:54.761228    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" event={"ID":"5a7348fd-346f-4b77-a11d-2969029c5cf4","Type":"ContainerStarted","Data":"89e8ddfe1a3a8081bfe7bd8d3e89d855cf45daf21147b64b5f8e33b22d63e282"}
Oct 30 21:21:54.800316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:54.800267    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podStartSLOduration=1.8002552459999999 podStartE2EDuration="1.800255246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:54.799281666 +0000 UTC m=+697.721739964" watchObservedRunningTime="2026-10-30 21:21:54.800255246 +0000 UTC m=+697.722713543"
Oct 30 21:21:55.117013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:21:55.116980    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:22:01.072904 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:22:01.072869    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:22:04.117197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:04.117158    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:22:04.130308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:04.130279    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:22:04.854287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:04.854258    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:22:24.285760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.285553    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"]
Oct 30 21:22:24.290498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.290480    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.306592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.306565    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"]
Oct 30 21:22:24.316229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.316200    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"default-dockercfg-tn85b\""
Oct 30 21:22:24.316392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.316342    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"premium-w1-b31518-kserve-self-signed-certs\""
Oct 30 21:22:24.316392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.316352    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"openshift-service-ca.crt\""
Oct 30 21:22:24.316516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.316427    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b31518\"/\"kube-root-ca.crt\""
Oct 30 21:22:24.318809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.318783    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.318936 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.318820    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.318936 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.318859    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.319052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.319008    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.319143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.319129    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.319197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.319159    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mht\" (UniqueName: \"kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.420584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.420546    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.420584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.420585    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mht\" (UniqueName: \"kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.420880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.420687    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.420880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.420715    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.420880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.420746    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.420880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.420795    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.421166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.421144    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.421221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.421192    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.421221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.421213    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.423081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.423062    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.423214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.423198    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.439924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.439897    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mht\" (UniqueName: \"kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht\") pod \"premium-w1-b31518-kserve-b76879949-qmb86\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") " pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.604836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.604742    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:24.904693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.904567    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"]
Oct 30 21:22:24.909662 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:22:24.909621    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4616cf15_31ad_470b_823b_e99e2143d885.slice/crio-85c0fc0bee53606614073839fc14656627cf3d858e7802bad8ec51d36c178768 WatchSource:0}: Error finding container 85c0fc0bee53606614073839fc14656627cf3d858e7802bad8ec51d36c178768: Status 404 returned error can't find the container with id 85c0fc0bee53606614073839fc14656627cf3d858e7802bad8ec51d36c178768
Oct 30 21:22:24.952751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:24.952720    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" event={"ID":"4616cf15-31ad-470b-823b-e99e2143d885","Type":"ContainerStarted","Data":"85c0fc0bee53606614073839fc14656627cf3d858e7802bad8ec51d36c178768"}
Oct 30 21:22:25.959950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:25.959903    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" event={"ID":"4616cf15-31ad-470b-823b-e99e2143d885","Type":"ContainerStarted","Data":"048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34"}
Oct 30 21:22:26.604999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:26.604959    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:31.085716 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:22:31.085624    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:22:34.605154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:34.605123    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:34.617977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:34.617951    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:34.657903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:34.657858    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podStartSLOduration=10.657846857 podStartE2EDuration="10.657846857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:25.99725406 +0000 UTC m=+728.919712361" watchObservedRunningTime="2026-10-30 21:22:34.657846857 +0000 UTC m=+737.580305155"
Oct 30 21:22:35.022580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:35.022555    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:22:39.299618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.299580    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"]
Oct 30 21:22:39.303498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.303476    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.318620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.318206    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-50ab77-openshift-default-dockercfg-tt79b\""
Oct 30 21:22:39.331323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.331291    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"]
Oct 30 21:22:39.395000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.394956    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395032    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sq76\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395166    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395209    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395315    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395369    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395409    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395433    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.395581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.395565    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504256    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504321    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504371    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504416    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504455    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504498    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504572    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504664    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.504717    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sq76\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.505526    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.505792    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.509153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.506013    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.510710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.510680    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.513744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.511428    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.517107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.517059    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.526545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.526515    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.531512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.531481    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sq76\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.541638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.541587    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.627880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.627776    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:39.951484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.951394    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"]
Oct 30 21:22:39.955294 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:22:39.955267    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6a72a5_8ccb_4ec9_9d3b_bff162c10bf6.slice/crio-03440bcb287876623a5c87efb45c6240ff97a72760030102769ddf073b0f5878 WatchSource:0}: Error finding container 03440bcb287876623a5c87efb45c6240ff97a72760030102769ddf073b0f5878: Status 404 returned error can't find the container with id 03440bcb287876623a5c87efb45c6240ff97a72760030102769ddf073b0f5878
Oct 30 21:22:39.958020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.957989    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:22:39.958115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.958052    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:22:39.958115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:39.958085    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:22:40.041361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:40.041333    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" event={"ID":"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6","Type":"ContainerStarted","Data":"84475e03713aee11cc4cae482f5aaee9ff39d298bc9876b2d78f6ea1a4702445"}
Oct 30 21:22:40.041488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:40.041391    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" event={"ID":"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6","Type":"ContainerStarted","Data":"03440bcb287876623a5c87efb45c6240ff97a72760030102769ddf073b0f5878"}
Oct 30 21:22:40.085667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:40.085597    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podStartSLOduration=1.085582602 podStartE2EDuration="1.085582602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:40.084661228 +0000 UTC m=+743.007119529" watchObservedRunningTime="2026-10-30 21:22:40.085582602 +0000 UTC m=+743.008040926"
Oct 30 21:22:40.636685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:40.630443    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:41.632619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:41.632563    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:41.632910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:41.632879    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:22:42.629028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:42.628989    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:42.629725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:42.629693    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:22:43.631009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:43.630971    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:43.631588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:43.631556    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:22:44.629020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:44.628979    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:44.629361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:44.629320    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:22:45.632628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:45.631964    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:45.632628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:45.632037    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:22:46.802156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:46.800362    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:46.802156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:46.801593    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:46.805070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:46.805047    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:22:48.675628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.672558    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"]
Oct 30 21:22:48.727858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.725957    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.753412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.751123    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:48.753412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.751869    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:22:48.753412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.751921    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:48.753412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.751974    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:48.766630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.766363    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"]
Oct 30 21:22:48.878196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.878161    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.878374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.878227    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.878374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.878362    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8rw\" (UniqueName: \"kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.979711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.979580    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.979890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.979712    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.979890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.979828    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8rw\" (UniqueName: \"kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.980469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.980442    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.982238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.982206    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:48.998526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:48.998498    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8rw\" (UniqueName: \"kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw\") pod \"maas-api-revoke-sub-9aad469c-6jnsl\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:49.069001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:49.068959    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:49.371354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:49.371281    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"]
Oct 30 21:22:49.375187 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:22:49.375161    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode838aae8_cad3_46aa_9aad_a7f8813d0879.slice/crio-f09613211b7a6ee28d8242e28b2555f4ef1a1cb746cad5c49118668a5b7c0d5c WatchSource:0}: Error finding container f09613211b7a6ee28d8242e28b2555f4ef1a1cb746cad5c49118668a5b7c0d5c: Status 404 returned error can't find the container with id f09613211b7a6ee28d8242e28b2555f4ef1a1cb746cad5c49118668a5b7c0d5c
Oct 30 21:22:50.104684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.104647    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl" event={"ID":"e838aae8-cad3-46aa-9aad-a7f8813d0879","Type":"ContainerStarted","Data":"fe69f80a8f64d6e6c3e20f26599dc9dc97cda7758688ffb05af72a7e64d1e054"}
Oct 30 21:22:50.105119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.104701    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl" event={"ID":"e838aae8-cad3-46aa-9aad-a7f8813d0879","Type":"ContainerStarted","Data":"f09613211b7a6ee28d8242e28b2555f4ef1a1cb746cad5c49118668a5b7c0d5c"}
Oct 30 21:22:50.846836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.846760    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl" podStartSLOduration=2.846721605 podStartE2EDuration="2.846721605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.140853423 +0000 UTC m=+753.063311720" watchObservedRunningTime="2026-10-30 21:22:50.846721605 +0000 UTC m=+753.769179904"
Oct 30 21:22:50.849716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.849689    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"]
Oct 30 21:22:50.854127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.854108    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:50.870006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.869982    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"]
Oct 30 21:22:51.000023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:50.999987    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27nqg\" (UniqueName: \"kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.000220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.000175    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.000283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.000233    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.101156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.101072    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27nqg\" (UniqueName: \"kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.101318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.101179    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.101318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.101207    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.101816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.101793    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.103618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.103583    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.111716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.111690    6695 generic.go:356] "Generic (PLEG): container finished" podID="e838aae8-cad3-46aa-9aad-a7f8813d0879" containerID="fe69f80a8f64d6e6c3e20f26599dc9dc97cda7758688ffb05af72a7e64d1e054" exitCode=0
Oct 30 21:22:51.112034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.111781    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl" event={"ID":"e838aae8-cad3-46aa-9aad-a7f8813d0879","Type":"ContainerDied","Data":"fe69f80a8f64d6e6c3e20f26599dc9dc97cda7758688ffb05af72a7e64d1e054"}
Oct 30 21:22:51.119151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.119130    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27nqg\" (UniqueName: \"kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg\") pod \"maas-api-revoke-sub-238dcb05-hc87t\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.167766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.167736    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:51.473677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:51.473571    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"]
Oct 30 21:22:51.478101 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:22:51.478073    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ae3e32_0b1f_4b79_8ad6_e00a32aff5ac.slice/crio-a7da041cc95f8170521a4d5081b4324d3d1aa44567542b9c2be1b54485b26ca2 WatchSource:0}: Error finding container a7da041cc95f8170521a4d5081b4324d3d1aa44567542b9c2be1b54485b26ca2: Status 404 returned error can't find the container with id a7da041cc95f8170521a4d5081b4324d3d1aa44567542b9c2be1b54485b26ca2
Oct 30 21:22:52.119213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.119179    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t" event={"ID":"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac","Type":"ContainerStarted","Data":"b25ce73f472bf6be8ec83be98c554136dfd50175b19f48a372982a24bf00a38f"}
Oct 30 21:22:52.119658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.119234    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t" event={"ID":"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac","Type":"ContainerStarted","Data":"a7da041cc95f8170521a4d5081b4324d3d1aa44567542b9c2be1b54485b26ca2"}
Oct 30 21:22:52.158145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.158062    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t" podStartSLOduration=2.158042757 podStartE2EDuration="2.158042757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:52.156112304 +0000 UTC m=+755.078570602" watchObservedRunningTime="2026-10-30 21:22:52.158042757 +0000 UTC m=+755.080501052"
Oct 30 21:22:52.267493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.267464    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:52.407721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.407636    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"]
Oct 30 21:22:52.408161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.408148    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e838aae8-cad3-46aa-9aad-a7f8813d0879" containerName="revoke-keys"
Oct 30 21:22:52.408199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.408163    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e838aae8-cad3-46aa-9aad-a7f8813d0879" containerName="revoke-keys"
Oct 30 21:22:52.408273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.408264    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e838aae8-cad3-46aa-9aad-a7f8813d0879" containerName="revoke-keys"
Oct 30 21:22:52.411448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.411426    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.416297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.416269    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw\" (UniqueName: \"kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw\") pod \"e838aae8-cad3-46aa-9aad-a7f8813d0879\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") "
Oct 30 21:22:52.416420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.416362    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca\") pod \"e838aae8-cad3-46aa-9aad-a7f8813d0879\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") "
Oct 30 21:22:52.416525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.416507    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca\") pod \"e838aae8-cad3-46aa-9aad-a7f8813d0879\" (UID: \"e838aae8-cad3-46aa-9aad-a7f8813d0879\") "
Oct 30 21:22:52.416910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.416874    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca" pod "e838aae8-cad3-46aa-9aad-a7f8813d0879" (UID: "e838aae8-cad3-46aa-9aad-a7f8813d0879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:52.417028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.416977    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:52.418276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.418253    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw" pod "e838aae8-cad3-46aa-9aad-a7f8813d0879" (UID: "e838aae8-cad3-46aa-9aad-a7f8813d0879"). InnerVolumeSpecName "kube-api-access-pg8rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:52.418364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.418352    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca" pod "e838aae8-cad3-46aa-9aad-a7f8813d0879" (UID: "e838aae8-cad3-46aa-9aad-a7f8813d0879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:52.428868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.428842    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"]
Oct 30 21:22:52.518223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.518189    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.518406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.518389    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.518465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.518415    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj94m\" (UniqueName: \"kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.518501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.518462    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e838aae8-cad3-46aa-9aad-a7f8813d0879-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:52.518543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.518494    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8rw\" (UniqueName: \"kubernetes.io/projected/e838aae8-cad3-46aa-9aad-a7f8813d0879-kube-api-access-pg8rw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:52.622888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.620900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.622888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.620946    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj94m\" (UniqueName: \"kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.622888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.621001    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.622888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.621982    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.624434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.624383    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.640865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.640838    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj94m\" (UniqueName: \"kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m\") pod \"maas-api-revoke-sub-4cd37886-sk77x\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:52.767833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:52.767751    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:53.064055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.064021    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"]
Oct 30 21:22:53.067876 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:22:53.067850    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c21f2c_8b1e_4af4_8b39_390237da2653.slice/crio-30e9fe83186b5bca22bb3ea2e35405db2580f6a5f129bb075bd46ed4ad50596e WatchSource:0}: Error finding container 30e9fe83186b5bca22bb3ea2e35405db2580f6a5f129bb075bd46ed4ad50596e: Status 404 returned error can't find the container with id 30e9fe83186b5bca22bb3ea2e35405db2580f6a5f129bb075bd46ed4ad50596e
Oct 30 21:22:53.132086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.132060    6695 generic.go:356] "Generic (PLEG): container finished" podID="17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" containerID="b25ce73f472bf6be8ec83be98c554136dfd50175b19f48a372982a24bf00a38f" exitCode=0
Oct 30 21:22:53.132467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.132131    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t" event={"ID":"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac","Type":"ContainerDied","Data":"b25ce73f472bf6be8ec83be98c554136dfd50175b19f48a372982a24bf00a38f"}
Oct 30 21:22:53.133476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.133456    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"
Oct 30 21:22:53.133476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.133463    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl" event={"ID":"e838aae8-cad3-46aa-9aad-a7f8813d0879","Type":"ContainerDied","Data":"f09613211b7a6ee28d8242e28b2555f4ef1a1cb746cad5c49118668a5b7c0d5c"}
Oct 30 21:22:53.133655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.133500    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f09613211b7a6ee28d8242e28b2555f4ef1a1cb746cad5c49118668a5b7c0d5c"
Oct 30 21:22:53.134784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:53.134761    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x" event={"ID":"d8c21f2c-8b1e-4af4-8b39-390237da2653","Type":"ContainerStarted","Data":"30e9fe83186b5bca22bb3ea2e35405db2580f6a5f129bb075bd46ed4ad50596e"}
Oct 30 21:22:54.142222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.142125    6695 generic.go:356] "Generic (PLEG): container finished" podID="d8c21f2c-8b1e-4af4-8b39-390237da2653" containerID="3f72f09aab445d3f87c049cf08757e1335c38d240b9efcafe0d24484781cf47c" exitCode=0
Oct 30 21:22:54.142632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.142227    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x" event={"ID":"d8c21f2c-8b1e-4af4-8b39-390237da2653","Type":"ContainerDied","Data":"3f72f09aab445d3f87c049cf08757e1335c38d240b9efcafe0d24484781cf47c"}
Oct 30 21:22:54.302275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.302252    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:54.336775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.336746    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca\") pod \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") "
Oct 30 21:22:54.336935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.336921    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg\" (UniqueName: \"kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg\") pod \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") "
Oct 30 21:22:54.336980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.336944    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca\") pod \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\" (UID: \"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac\") "
Oct 30 21:22:54.337101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.337080    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca" pod "17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" (UID: "17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:54.337272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.337254    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:54.338884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.338861    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg" pod "17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" (UID: "17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac"). InnerVolumeSpecName "kube-api-access-27nqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:54.338952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.338889    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca" pod "17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" (UID: "17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:54.438057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.437984    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27nqg\" (UniqueName: \"kubernetes.io/projected/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-kube-api-access-27nqg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:54.438057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:54.438011    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:55.149594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.149564    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"
Oct 30 21:22:55.150046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.149612    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t" event={"ID":"17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac","Type":"ContainerDied","Data":"a7da041cc95f8170521a4d5081b4324d3d1aa44567542b9c2be1b54485b26ca2"}
Oct 30 21:22:55.150046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.149669    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7da041cc95f8170521a4d5081b4324d3d1aa44567542b9c2be1b54485b26ca2"
Oct 30 21:22:55.285631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.285591    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:55.345491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.345457    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca\") pod \"d8c21f2c-8b1e-4af4-8b39-390237da2653\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") "
Oct 30 21:22:55.345678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.345558    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m\" (UniqueName: \"kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m\") pod \"d8c21f2c-8b1e-4af4-8b39-390237da2653\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") "
Oct 30 21:22:55.345678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.345630    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca\") pod \"d8c21f2c-8b1e-4af4-8b39-390237da2653\" (UID: \"d8c21f2c-8b1e-4af4-8b39-390237da2653\") "
Oct 30 21:22:55.345979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.345948    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca" pod "d8c21f2c-8b1e-4af4-8b39-390237da2653" (UID: "d8c21f2c-8b1e-4af4-8b39-390237da2653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:55.347674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.347645    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca" pod "d8c21f2c-8b1e-4af4-8b39-390237da2653" (UID: "d8c21f2c-8b1e-4af4-8b39-390237da2653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:55.347674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.347660    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m" pod "d8c21f2c-8b1e-4af4-8b39-390237da2653" (UID: "d8c21f2c-8b1e-4af4-8b39-390237da2653"). InnerVolumeSpecName "kube-api-access-dj94m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:55.446844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.446760    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:55.446844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.446798    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj94m\" (UniqueName: \"kubernetes.io/projected/d8c21f2c-8b1e-4af4-8b39-390237da2653-kube-api-access-dj94m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:55.446844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:55.446815    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8c21f2c-8b1e-4af4-8b39-390237da2653-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:22:56.155445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:56.155412    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"
Oct 30 21:22:56.155823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:56.155442    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x" event={"ID":"d8c21f2c-8b1e-4af4-8b39-390237da2653","Type":"ContainerDied","Data":"30e9fe83186b5bca22bb3ea2e35405db2580f6a5f129bb075bd46ed4ad50596e"}
Oct 30 21:22:56.155823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:56.155491    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e9fe83186b5bca22bb3ea2e35405db2580f6a5f129bb075bd46ed4ad50596e"
Oct 30 21:22:57.838636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.838584    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"]
Oct 30 21:22:57.839089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.839073    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d8c21f2c-8b1e-4af4-8b39-390237da2653" containerName="revoke-keys"
Oct 30 21:22:57.839128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.839091    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c21f2c-8b1e-4af4-8b39-390237da2653" containerName="revoke-keys"
Oct 30 21:22:57.839128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.839103    6695 cpu_manager.go:414] "containerMap: removing container" podUID="17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" containerName="revoke-keys"
Oct 30 21:22:57.839128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.839108    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" containerName="revoke-keys"
Oct 30 21:22:57.839225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.839215    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c21f2c-8b1e-4af4-8b39-390237da2653" containerName="revoke-keys"
Oct 30 21:22:57.839256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.839227    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" containerName="revoke-keys"
Oct 30 21:22:57.842795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.842777    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.861692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.861667    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"]
Oct 30 21:22:57.870648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.870624    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:57.870648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.870644    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:57.870856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.870685    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:57.870856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.870765    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:22:57.893001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.892975    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.893138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.893102    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.893179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.893159    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnnw\" (UniqueName: \"kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.994693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.994660    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.994839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.994722    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnnw\" (UniqueName: \"kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.994839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.994774    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.995519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.995492    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:57.996938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:57.996912    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:58.013689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:58.013664    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnnw\" (UniqueName: \"kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw\") pod \"maas-api-revoke-sub-ebd4fd4e-sm6jq\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:58.156676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:58.156553    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:22:58.452647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:58.452539    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"]
Oct 30 21:22:58.456119 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:22:58.456092    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50211ceb_5c13_4dc7_ab8b_44a2eba16e04.slice/crio-24aef3a0735ac877121fbdf5739b429936eeaff3c38f12e2756f1308982bb354 WatchSource:0}: Error finding container 24aef3a0735ac877121fbdf5739b429936eeaff3c38f12e2756f1308982bb354: Status 404 returned error can't find the container with id 24aef3a0735ac877121fbdf5739b429936eeaff3c38f12e2756f1308982bb354
Oct 30 21:22:59.176431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:59.176397    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq" event={"ID":"50211ceb-5c13-4dc7-ab8b-44a2eba16e04","Type":"ContainerStarted","Data":"e19d9e02e06733779528ee88c185ea06cf3153e3fd2c847fbd8c4943148378bc"}
Oct 30 21:22:59.176826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:59.176449    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq" event={"ID":"50211ceb-5c13-4dc7-ab8b-44a2eba16e04","Type":"ContainerStarted","Data":"24aef3a0735ac877121fbdf5739b429936eeaff3c38f12e2756f1308982bb354"}
Oct 30 21:22:59.214045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:22:59.213999    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq" podStartSLOduration=2.213986395 podStartE2EDuration="2.213986395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:59.212564288 +0000 UTC m=+762.135022582" watchObservedRunningTime="2026-10-30 21:22:59.213986395 +0000 UTC m=+762.136444730"
Oct 30 21:23:00.183080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:00.183046    6695 generic.go:356] "Generic (PLEG): container finished" podID="50211ceb-5c13-4dc7-ab8b-44a2eba16e04" containerID="e19d9e02e06733779528ee88c185ea06cf3153e3fd2c847fbd8c4943148378bc" exitCode=0
Oct 30 21:23:00.183466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:00.183130    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq" event={"ID":"50211ceb-5c13-4dc7-ab8b-44a2eba16e04","Type":"ContainerDied","Data":"e19d9e02e06733779528ee88c185ea06cf3153e3fd2c847fbd8c4943148378bc"}
Oct 30 21:23:01.077473 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:23:01.077435    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:23:01.356022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.355958    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:23:01.441397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.441361    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca\") pod \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") "
Oct 30 21:23:01.441563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.441413    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca\") pod \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") "
Oct 30 21:23:01.441563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.441460    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw\" (UniqueName: \"kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw\") pod \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\" (UID: \"50211ceb-5c13-4dc7-ab8b-44a2eba16e04\") "
Oct 30 21:23:01.441814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.441785    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca" pod "50211ceb-5c13-4dc7-ab8b-44a2eba16e04" (UID: "50211ceb-5c13-4dc7-ab8b-44a2eba16e04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:01.443517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.443487    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca" pod "50211ceb-5c13-4dc7-ab8b-44a2eba16e04" (UID: "50211ceb-5c13-4dc7-ab8b-44a2eba16e04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:01.443651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.443552    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw" pod "50211ceb-5c13-4dc7-ab8b-44a2eba16e04" (UID: "50211ceb-5c13-4dc7-ab8b-44a2eba16e04"). InnerVolumeSpecName "kube-api-access-cpnnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:01.543327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.543290    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnnw\" (UniqueName: \"kubernetes.io/projected/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-kube-api-access-cpnnw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:01.543327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.543322    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:01.543327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:01.543332    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50211ceb-5c13-4dc7-ab8b-44a2eba16e04-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:02.197371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:02.197335    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"
Oct 30 21:23:02.197622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:02.197370    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq" event={"ID":"50211ceb-5c13-4dc7-ab8b-44a2eba16e04","Type":"ContainerDied","Data":"24aef3a0735ac877121fbdf5739b429936eeaff3c38f12e2756f1308982bb354"}
Oct 30 21:23:02.197622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:02.197472    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24aef3a0735ac877121fbdf5739b429936eeaff3c38f12e2756f1308982bb354"
Oct 30 21:23:03.772383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.772349    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"]
Oct 30 21:23:03.774201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.774172    6695 cpu_manager.go:414] "containerMap: removing container" podUID="50211ceb-5c13-4dc7-ab8b-44a2eba16e04" containerName="revoke-keys"
Oct 30 21:23:03.774381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.774361    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50211ceb-5c13-4dc7-ab8b-44a2eba16e04" containerName="revoke-keys"
Oct 30 21:23:03.774767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.774737    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="50211ceb-5c13-4dc7-ab8b-44a2eba16e04" containerName="revoke-keys"
Oct 30 21:23:03.782565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.782539    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.792268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.792130    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"]
Oct 30 21:23:03.801792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.801756    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:23:03.802355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.802333    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:03.802476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.802426    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:03.813871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.813842    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:03.874400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.874365    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.874585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.874426    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.874585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.874546    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkhp\" (UniqueName: \"kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.975647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.975590    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.975824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.975705    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.975824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.975788    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkhp\" (UniqueName: \"kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.976325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.976298    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:03.978300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:03.978275    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:04.019625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:04.018925    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkhp\" (UniqueName: \"kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp\") pod \"maas-api-revoke-sub-7e79c2e8-5s5zw\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:04.121773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:04.121693    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:04.421026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:04.420942    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"]
Oct 30 21:23:04.425082 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:04.425051    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48ca0a4_e6b1_4d32_8f7e_459be7368682.slice/crio-a2805fac2efed04e3a49a2f6cef10eff1a4be1decb85872f58d2ae2cf945934f WatchSource:0}: Error finding container a2805fac2efed04e3a49a2f6cef10eff1a4be1decb85872f58d2ae2cf945934f: Status 404 returned error can't find the container with id a2805fac2efed04e3a49a2f6cef10eff1a4be1decb85872f58d2ae2cf945934f
Oct 30 21:23:05.216891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:05.216850    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw" event={"ID":"a48ca0a4-e6b1-4d32-8f7e-459be7368682","Type":"ContainerStarted","Data":"ac859775beecee786a8a734d3e16380d5ecbd6dd082b9ac42988c583dfbaf286"}
Oct 30 21:23:05.217288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:05.216918    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw" event={"ID":"a48ca0a4-e6b1-4d32-8f7e-459be7368682","Type":"ContainerStarted","Data":"a2805fac2efed04e3a49a2f6cef10eff1a4be1decb85872f58d2ae2cf945934f"}
Oct 30 21:23:05.256045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:05.255867    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw" podStartSLOduration=2.255854764 podStartE2EDuration="2.255854764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:05.254047733 +0000 UTC m=+768.176506032" watchObservedRunningTime="2026-10-30 21:23:05.255854764 +0000 UTC m=+768.178313061"
Oct 30 21:23:06.223656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.223589    6695 generic.go:356] "Generic (PLEG): container finished" podID="a48ca0a4-e6b1-4d32-8f7e-459be7368682" containerID="ac859775beecee786a8a734d3e16380d5ecbd6dd082b9ac42988c583dfbaf286" exitCode=0
Oct 30 21:23:06.224070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.223680    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw" event={"ID":"a48ca0a4-e6b1-4d32-8f7e-459be7368682","Type":"ContainerDied","Data":"ac859775beecee786a8a734d3e16380d5ecbd6dd082b9ac42988c583dfbaf286"}
Oct 30 21:23:06.828791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.828753    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"]
Oct 30 21:23:06.832303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.832285    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:06.849112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.849086    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"]
Oct 30 21:23:06.906265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.906233    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntv7j\" (UniqueName: \"kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:06.906428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.906396    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:06.906472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:06.906460    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.007836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.007798    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.007974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.007852    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.007974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.007913    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntv7j\" (UniqueName: \"kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.008440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.008418    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.010192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.010176    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.026636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.026595    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntv7j\" (UniqueName: \"kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j\") pod \"maas-api-revoke-sub-c91d102c-k2jzt\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.146256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.146170    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:07.365966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.365941    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:07.458852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.458816    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"]
Oct 30 21:23:07.460639 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:07.459887    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d808b5_0424_4d9e_bada_5634c4713e8f.slice/crio-820720ef479fd0eeaf02fab4a1fd113a195726c7adebceb861d2c8f4f2337e71 WatchSource:0}: Error finding container 820720ef479fd0eeaf02fab4a1fd113a195726c7adebceb861d2c8f4f2337e71: Status 404 returned error can't find the container with id 820720ef479fd0eeaf02fab4a1fd113a195726c7adebceb861d2c8f4f2337e71
Oct 30 21:23:07.518091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.518064    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca\") pod \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") "
Oct 30 21:23:07.518226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.518178    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp\" (UniqueName: \"kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp\") pod \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") "
Oct 30 21:23:07.518267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.518232    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca\") pod \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\" (UID: \"a48ca0a4-e6b1-4d32-8f7e-459be7368682\") "
Oct 30 21:23:07.518565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.518541    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca" pod "a48ca0a4-e6b1-4d32-8f7e-459be7368682" (UID: "a48ca0a4-e6b1-4d32-8f7e-459be7368682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:07.519967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.519945    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca" pod "a48ca0a4-e6b1-4d32-8f7e-459be7368682" (UID: "a48ca0a4-e6b1-4d32-8f7e-459be7368682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:07.520058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.520040    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp" pod "a48ca0a4-e6b1-4d32-8f7e-459be7368682" (UID: "a48ca0a4-e6b1-4d32-8f7e-459be7368682"). InnerVolumeSpecName "kube-api-access-ngkhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:07.620068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.620039    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkhp\" (UniqueName: \"kubernetes.io/projected/a48ca0a4-e6b1-4d32-8f7e-459be7368682-kube-api-access-ngkhp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.620068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.620064    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:07.620068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:07.620073    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48ca0a4-e6b1-4d32-8f7e-459be7368682-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:08.238260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.238217    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"
Oct 30 21:23:08.238445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.238292    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw" event={"ID":"a48ca0a4-e6b1-4d32-8f7e-459be7368682","Type":"ContainerDied","Data":"a2805fac2efed04e3a49a2f6cef10eff1a4be1decb85872f58d2ae2cf945934f"}
Oct 30 21:23:08.238445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.238348    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2805fac2efed04e3a49a2f6cef10eff1a4be1decb85872f58d2ae2cf945934f"
Oct 30 21:23:08.239957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.239930    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt" event={"ID":"e9d808b5-0424-4d9e-bada-5634c4713e8f","Type":"ContainerStarted","Data":"7e5f783fe276c853f00da45997960a64ef2f9983a19ec94c8580d9d383540c64"}
Oct 30 21:23:08.240069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.239976    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt" event={"ID":"e9d808b5-0424-4d9e-bada-5634c4713e8f","Type":"ContainerStarted","Data":"820720ef479fd0eeaf02fab4a1fd113a195726c7adebceb861d2c8f4f2337e71"}
Oct 30 21:23:08.278987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.278896    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt" podStartSLOduration=2.27888233 podStartE2EDuration="2.27888233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:08.276907681 +0000 UTC m=+771.199365977" watchObservedRunningTime="2026-10-30 21:23:08.27888233 +0000 UTC m=+771.201340626"
Oct 30 21:23:08.536224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.536137    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"]
Oct 30 21:23:08.536668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.536653    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a48ca0a4-e6b1-4d32-8f7e-459be7368682" containerName="revoke-keys"
Oct 30 21:23:08.536738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.536670    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48ca0a4-e6b1-4d32-8f7e-459be7368682" containerName="revoke-keys"
Oct 30 21:23:08.536800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.536791    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48ca0a4-e6b1-4d32-8f7e-459be7368682" containerName="revoke-keys"
Oct 30 21:23:08.540200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.540176    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.555941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.555914    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"]
Oct 30 21:23:08.636888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.634580    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzsp\" (UniqueName: \"kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.636888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.634721    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.636888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.634801    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.736132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.736085    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.736315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.736211    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzsp\" (UniqueName: \"kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.736315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.736264    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.736919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.736892    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.738704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.738673    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.755530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.755495    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzsp\" (UniqueName: \"kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp\") pod \"maas-api-revoke-sub-0bdc6af8-4sk74\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:08.854958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:08.854868    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:09.168587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:09.162408    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"]
Oct 30 21:23:09.169739 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:09.169704    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f22cd8a_962a_4439_bd6c_f9aaf4cb386e.slice/crio-2af9e226e8e49b452aae5d7d67be728d26d8719bf55a302bf14db8b2728f76d6 WatchSource:0}: Error finding container 2af9e226e8e49b452aae5d7d67be728d26d8719bf55a302bf14db8b2728f76d6: Status 404 returned error can't find the container with id 2af9e226e8e49b452aae5d7d67be728d26d8719bf55a302bf14db8b2728f76d6
Oct 30 21:23:09.253541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:09.253510    6695 generic.go:356] "Generic (PLEG): container finished" podID="e9d808b5-0424-4d9e-bada-5634c4713e8f" containerID="7e5f783fe276c853f00da45997960a64ef2f9983a19ec94c8580d9d383540c64" exitCode=0
Oct 30 21:23:09.253718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:09.253634    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt" event={"ID":"e9d808b5-0424-4d9e-bada-5634c4713e8f","Type":"ContainerDied","Data":"7e5f783fe276c853f00da45997960a64ef2f9983a19ec94c8580d9d383540c64"}
Oct 30 21:23:09.256638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:09.254859    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74" event={"ID":"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e","Type":"ContainerStarted","Data":"2af9e226e8e49b452aae5d7d67be728d26d8719bf55a302bf14db8b2728f76d6"}
Oct 30 21:23:10.262835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.262745    6695 generic.go:356] "Generic (PLEG): container finished" podID="0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" containerID="5d1903a5cd0fa9229fc0d6300590f6fdfc6e92549cc19e873709bca6d17ce532" exitCode=0
Oct 30 21:23:10.263225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.262833    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74" event={"ID":"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e","Type":"ContainerDied","Data":"5d1903a5cd0fa9229fc0d6300590f6fdfc6e92549cc19e873709bca6d17ce532"}
Oct 30 21:23:10.396071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.396050    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:10.560296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.560259    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca\") pod \"e9d808b5-0424-4d9e-bada-5634c4713e8f\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") "
Oct 30 21:23:10.560467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.560434    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca\") pod \"e9d808b5-0424-4d9e-bada-5634c4713e8f\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") "
Oct 30 21:23:10.560564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.560550    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j\" (UniqueName: \"kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j\") pod \"e9d808b5-0424-4d9e-bada-5634c4713e8f\" (UID: \"e9d808b5-0424-4d9e-bada-5634c4713e8f\") "
Oct 30 21:23:10.560639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.560566    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca" pod "e9d808b5-0424-4d9e-bada-5634c4713e8f" (UID: "e9d808b5-0424-4d9e-bada-5634c4713e8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:10.560903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.560887    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:10.562662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.562637    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca" pod "e9d808b5-0424-4d9e-bada-5634c4713e8f" (UID: "e9d808b5-0424-4d9e-bada-5634c4713e8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:10.562784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.562695    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j" pod "e9d808b5-0424-4d9e-bada-5634c4713e8f" (UID: "e9d808b5-0424-4d9e-bada-5634c4713e8f"). InnerVolumeSpecName "kube-api-access-ntv7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:10.662218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.662190    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntv7j\" (UniqueName: \"kubernetes.io/projected/e9d808b5-0424-4d9e-bada-5634c4713e8f-kube-api-access-ntv7j\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:10.662218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:10.662216    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d808b5-0424-4d9e-bada-5634c4713e8f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:11.269706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.269670    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt" event={"ID":"e9d808b5-0424-4d9e-bada-5634c4713e8f","Type":"ContainerDied","Data":"820720ef479fd0eeaf02fab4a1fd113a195726c7adebceb861d2c8f4f2337e71"}
Oct 30 21:23:11.269706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.269693    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"
Oct 30 21:23:11.270130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.269724    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820720ef479fd0eeaf02fab4a1fd113a195726c7adebceb861d2c8f4f2337e71"
Oct 30 21:23:11.411368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.411342    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:11.572217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.572188    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp\" (UniqueName: \"kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp\") pod \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") "
Oct 30 21:23:11.572402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.572307    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca\") pod \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") "
Oct 30 21:23:11.572402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.572360    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca\") pod \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\" (UID: \"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e\") "
Oct 30 21:23:11.572717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.572688    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca" pod "0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" (UID: "0f22cd8a-962a-4439-bd6c-f9aaf4cb386e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:11.574262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.574239    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp" pod "0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" (UID: "0f22cd8a-962a-4439-bd6c-f9aaf4cb386e"). InnerVolumeSpecName "kube-api-access-6gzsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:11.574400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.574379    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca" pod "0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" (UID: "0f22cd8a-962a-4439-bd6c-f9aaf4cb386e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:11.673389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.673356    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:11.673389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.673385    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:11.673389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:11.673394    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzsp\" (UniqueName: \"kubernetes.io/projected/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e-kube-api-access-6gzsp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:12.276886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:12.276857    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"
Oct 30 21:23:12.277355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:12.276860    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74" event={"ID":"0f22cd8a-962a-4439-bd6c-f9aaf4cb386e","Type":"ContainerDied","Data":"2af9e226e8e49b452aae5d7d67be728d26d8719bf55a302bf14db8b2728f76d6"}
Oct 30 21:23:12.277355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:12.276981    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af9e226e8e49b452aae5d7d67be728d26d8719bf55a302bf14db8b2728f76d6"
Oct 30 21:23:18.864246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864213    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"]
Oct 30 21:23:18.864722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864707    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" containerName="revoke-keys"
Oct 30 21:23:18.864782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864723    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" containerName="revoke-keys"
Oct 30 21:23:18.864782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864740    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e9d808b5-0424-4d9e-bada-5634c4713e8f" containerName="revoke-keys"
Oct 30 21:23:18.864782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864746    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d808b5-0424-4d9e-bada-5634c4713e8f" containerName="revoke-keys"
Oct 30 21:23:18.864877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864825    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" containerName="revoke-keys"
Oct 30 21:23:18.864877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.864835    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d808b5-0424-4d9e-bada-5634c4713e8f" containerName="revoke-keys"
Oct 30 21:23:18.869183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.869151    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:18.886227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.886202    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"]
Oct 30 21:23:18.894512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.894490    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:18.894671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.894631    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:18.895157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.895139    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:23:18.895254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.895161    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:18.899671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.899646    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhp5\" (UniqueName: \"kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:18.899775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.899735    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:18.899852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:18.899800    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.000424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.000384    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.000591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.000464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.000591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.000559    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhp5\" (UniqueName: \"kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.001106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.001086    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.002826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.002797    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.019300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.019271    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhp5\" (UniqueName: \"kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5\") pod \"maas-api-revoke-sub-f58832e5-xcdjf\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.183526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.183452    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:19.482895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:19.482810    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"]
Oct 30 21:23:19.486468 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:19.486440    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f70c919_d28e_4ba0_90b3_f79e829bc93a.slice/crio-193fff686c86f455610ca4f62ccdef67437a29e7b0ec2d0ed616cbe6c3278601 WatchSource:0}: Error finding container 193fff686c86f455610ca4f62ccdef67437a29e7b0ec2d0ed616cbe6c3278601: Status 404 returned error can't find the container with id 193fff686c86f455610ca4f62ccdef67437a29e7b0ec2d0ed616cbe6c3278601
Oct 30 21:23:20.382098 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:23:20.382055    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f70c919_d28e_4ba0_90b3_f79e829bc93a.slice/crio-9f5f5ac01e4ba1ff8bc536cf1e1c809f76cfddd2e7db92cb94e9f1f4b86a5813.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f70c919_d28e_4ba0_90b3_f79e829bc93a.slice/crio-conmon-9f5f5ac01e4ba1ff8bc536cf1e1c809f76cfddd2e7db92cb94e9f1f4b86a5813.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:20.443268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:20.443236    6695 generic.go:356] "Generic (PLEG): container finished" podID="7f70c919-d28e-4ba0-90b3-f79e829bc93a" containerID="9f5f5ac01e4ba1ff8bc536cf1e1c809f76cfddd2e7db92cb94e9f1f4b86a5813" exitCode=0
Oct 30 21:23:20.443432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:20.443319    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf" event={"ID":"7f70c919-d28e-4ba0-90b3-f79e829bc93a","Type":"ContainerDied","Data":"9f5f5ac01e4ba1ff8bc536cf1e1c809f76cfddd2e7db92cb94e9f1f4b86a5813"}
Oct 30 21:23:20.443432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:20.443373    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf" event={"ID":"7f70c919-d28e-4ba0-90b3-f79e829bc93a","Type":"ContainerStarted","Data":"193fff686c86f455610ca4f62ccdef67437a29e7b0ec2d0ed616cbe6c3278601"}
Oct 30 21:23:21.594658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.594633    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:21.628283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.628248    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca\") pod \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") "
Oct 30 21:23:21.628426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.628309    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca\") pod \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") "
Oct 30 21:23:21.628426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.628400    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5\" (UniqueName: \"kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5\") pod \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\" (UID: \"7f70c919-d28e-4ba0-90b3-f79e829bc93a\") "
Oct 30 21:23:21.628691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.628669    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca" pod "7f70c919-d28e-4ba0-90b3-f79e829bc93a" (UID: "7f70c919-d28e-4ba0-90b3-f79e829bc93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:21.628872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.628858    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:21.630439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.630392    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca" pod "7f70c919-d28e-4ba0-90b3-f79e829bc93a" (UID: "7f70c919-d28e-4ba0-90b3-f79e829bc93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:21.630554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.630515    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5" pod "7f70c919-d28e-4ba0-90b3-f79e829bc93a" (UID: "7f70c919-d28e-4ba0-90b3-f79e829bc93a"). InnerVolumeSpecName "kube-api-access-qwhp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:21.730265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.730184    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f70c919-d28e-4ba0-90b3-f79e829bc93a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:21.730265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:21.730217    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhp5\" (UniqueName: \"kubernetes.io/projected/7f70c919-d28e-4ba0-90b3-f79e829bc93a-kube-api-access-qwhp5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:23:22.455301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:22.455262    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf" event={"ID":"7f70c919-d28e-4ba0-90b3-f79e829bc93a","Type":"ContainerDied","Data":"193fff686c86f455610ca4f62ccdef67437a29e7b0ec2d0ed616cbe6c3278601"}
Oct 30 21:23:22.455480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:22.455312    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="193fff686c86f455610ca4f62ccdef67437a29e7b0ec2d0ed616cbe6c3278601"
Oct 30 21:23:22.455480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:22.455277    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"
Oct 30 21:23:23.813908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.813871    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"]
Oct 30 21:23:23.814576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.814554    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7f70c919-d28e-4ba0-90b3-f79e829bc93a" containerName="revoke-keys"
Oct 30 21:23:23.814685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.814578    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f70c919-d28e-4ba0-90b3-f79e829bc93a" containerName="revoke-keys"
Oct 30 21:23:23.814809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.814789    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f70c919-d28e-4ba0-90b3-f79e829bc93a" containerName="revoke-keys"
Oct 30 21:23:23.818767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.818741    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.850641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.849108    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"]
Oct 30 21:23:23.860852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.860800    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.860986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.860870    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5tf\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.860986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.860934    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.860986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.860964    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.861172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.860993    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.861172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.861028    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.861172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.861115    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.861322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.861172    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.861322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.861200    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.964240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.964204    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.964506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.964481    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.964642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.964626    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.964775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.964760    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.964951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.964935    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.966684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.966658    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.977111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.977085    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.977699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.977676    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.978276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.978252    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5tf\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.978837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.966080    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.978972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.977546    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.979059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.966276    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.979151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.966347    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:23.979238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:23.977001    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:24.004165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.004130    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:24.040229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.040196    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:24.044645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.041340    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:24.045428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.045346    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5tf\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:24.166203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.166104    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:24.561415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.561369    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"]
Oct 30 21:23:24.564400 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:24.564352    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecb1a6c_ecf4_42e3_bf15_1fc1a3fe87ae.slice/crio-1cb38db111ca7c5c40e6a9d368d5b483e6f5f76c4f11a04195adbd7d251f5255 WatchSource:0}: Error finding container 1cb38db111ca7c5c40e6a9d368d5b483e6f5f76c4f11a04195adbd7d251f5255: Status 404 returned error can't find the container with id 1cb38db111ca7c5c40e6a9d368d5b483e6f5f76c4f11a04195adbd7d251f5255
Oct 30 21:23:24.568642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.568115    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:24.568642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.568203    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:24.568642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.568248    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:24.935319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.935240    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"]
Oct 30 21:23:24.942803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.942773    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:24.964667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:24.964635    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"]
Oct 30 21:23:25.028305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028261    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028304    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028338    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028376    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028397    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028412    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrml\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028450    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.028541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.028503    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.129803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.129758    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.130515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.130487    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.130788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.130768    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131288    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131088    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrml\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131173    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.130418    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131393    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131443    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.131499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131489    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.132012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.131981    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.132320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.132300    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.139626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.136135    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.161625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.149485    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.164562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.164496    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.166162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.166137    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrml\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml\") pod \"maas-default-gateway-openshift-default-77f556d647-gzb2v\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.286181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.284887    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:25.547683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.546943    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" event={"ID":"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae","Type":"ContainerStarted","Data":"820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67"}
Oct 30 21:23:25.547683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.547014    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" event={"ID":"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae","Type":"ContainerStarted","Data":"1cb38db111ca7c5c40e6a9d368d5b483e6f5f76c4f11a04195adbd7d251f5255"}
Oct 30 21:23:25.613628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:25.607799    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podStartSLOduration=2.607776655 podStartE2EDuration="2.607776655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:25.592832284 +0000 UTC m=+788.515290582" watchObservedRunningTime="2026-10-30 21:23:25.607776655 +0000 UTC m=+788.530234954"
Oct 30 21:23:26.178025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.177991    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:26.191805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.191759    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:23:26.192080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.192031    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 21:23:26.317336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.274936    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"]
Oct 30 21:23:26.366858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.358060    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:26.366858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.358149    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:26.366858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.358213    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:26.705696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:26.705654    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" event={"ID":"19c587f1-3895-4032-ac4f-623b05bedcb2","Type":"ContainerStarted","Data":"e3796f4a7d30fd13a009de3b87ead08c19c703255041ced0d8943d769ada86b6"}
Oct 30 21:23:27.186627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:27.178073    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:23:27.186627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:27.178148    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 21:23:27.726504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:27.726459    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" event={"ID":"19c587f1-3895-4032-ac4f-623b05bedcb2","Type":"ContainerStarted","Data":"7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65"}
Oct 30 21:23:27.786662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:27.785478    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podStartSLOduration=3.7854604480000003 podStartE2EDuration="3.785460448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:27.785005225 +0000 UTC m=+790.707463522" watchObservedRunningTime="2026-10-30 21:23:27.785460448 +0000 UTC m=+790.707918746"
Oct 30 21:23:28.167623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:28.167580    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:23:28.167822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:28.167696    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 21:23:28.285815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:28.285772    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:28.287761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:28.287721    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:28.287918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:28.287781    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:29.167149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.167114    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:23:29.167405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.167172    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 21:23:29.286131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.286094    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:29.286512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.286149    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:29.757791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.757752    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"]
Oct 30 21:23:29.774778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.773165    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.785248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.785123    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"]
Oct 30 21:23:29.793901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.793876    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9d677d-openshift-default-dockercfg-l57n8\""
Oct 30 21:23:29.876317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876276    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876349    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876375    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876417    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5ph\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876490    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876542    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876586    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876641    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.876951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.876930    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.978708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.978661    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5ph\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.979356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.979327    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.980136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.980116    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.980364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.980047    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.980791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.980514    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.980966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.980723    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.981091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.980924    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.981227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.981212    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.981619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.981581    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.981941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.981923    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.982413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.982369    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.982564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.982548    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.982752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.982736    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.983637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.983425    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.983778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.983760    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.984308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.984288    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.997754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.997734    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5ph\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:29.999060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:29.999043    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:30.090017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.089981    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:30.171790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.171748    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:23:30.171981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.171813    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 21:23:30.286632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.286142    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:30.286632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.286202    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:30.489721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.489583    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"]
Oct 30 21:23:30.497732 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:30.497707    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66df6bb2_e479_47cf_bbc1_9e8ba3202637.slice/crio-bdc80895337536369bc334ade3776a962208d7c4f4381832d2b0c4c692d18688 WatchSource:0}: Error finding container bdc80895337536369bc334ade3776a962208d7c4f4381832d2b0c4c692d18688: Status 404 returned error can't find the container with id bdc80895337536369bc334ade3776a962208d7c4f4381832d2b0c4c692d18688
Oct 30 21:23:30.499240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.499208    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:30.499320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.499267    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:30.499320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.499295    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:30.782773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.782735    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" event={"ID":"66df6bb2-e479-47cf-bbc1-9e8ba3202637","Type":"ContainerStarted","Data":"615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb"}
Oct 30 21:23:30.783286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.783265    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" event={"ID":"66df6bb2-e479-47cf-bbc1-9e8ba3202637","Type":"ContainerStarted","Data":"bdc80895337536369bc334ade3776a962208d7c4f4381832d2b0c4c692d18688"}
Oct 30 21:23:30.849124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:30.849058    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podStartSLOduration=1.8490389870000001 podStartE2EDuration="1.849038987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:30.824838758 +0000 UTC m=+793.747297053" watchObservedRunningTime="2026-10-30 21:23:30.849038987 +0000 UTC m=+793.771497285"
Oct 30 21:23:31.077759 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:23:31.077674    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:23:31.090902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.090876    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:31.092624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.092576    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:31.092720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.092656    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:31.167401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.167363    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:23:31.167562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.167434    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 21:23:31.285772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.285735    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:31.285978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:31.285790    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:32.090532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.090502    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:32.091002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.090978    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:32.208622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.207844    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:32.216624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.209576    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:32.218911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.218841    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 21:23:32.312077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.312039    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:32.328768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:32.328714    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:33.102997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.102960    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:33.103566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.103532    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:33.288075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.288034    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:33.288359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.288327    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:33.713626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.710689    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:33.744628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.741742    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:33.852746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:33.852698    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:34.016705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.012683    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:34.091102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.091065    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:34.091411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.091365    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:34.300026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.299990    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:34.300584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.300555    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:34.711796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.711690    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:34.712466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.712440    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:23:34.751764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.751718    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:23:34.752081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.752052    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:34.866625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.861679    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:23:34.866625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:34.862313    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:23:35.064652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.063475    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:23:35.097425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.097386    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:35.097725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.097689    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:35.124895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.124849    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:35.471144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.471053    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:35.513709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.513665    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:35.864628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:35.863690    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:23:36.095628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:36.095488    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:36.095628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:36.095549    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:36.292403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:36.292367    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gzb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:23:36.292720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:36.292688    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 21:23:36.334974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:36.333857    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:23:37.090929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:37.090892    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:37.091324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:37.090954    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:38.091272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:38.091231    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:23:38.091896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:38.091860    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 21:23:38.240787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:38.240755    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:38.241127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:38.241101    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:38.242221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:38.242200    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:23:39.113455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:39.112553    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:39.118086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:39.114111    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:39.134770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:39.134732    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 21:23:42.572236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.572204    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"]
Oct 30 21:23:42.576984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.576961    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.603445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.603414    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-49738b-openshift-default-dockercfg-h4mdb\""
Oct 30 21:23:42.606909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.606879    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"]
Oct 30 21:23:42.653487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653449    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.653703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653493    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.653703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653640    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.653703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653681    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mgbg\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.653835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653804    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.653894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653876    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.653959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.653944    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.654025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.654007    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.654086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.654062    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755265    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755313    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755343    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755388    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755470    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755499    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755547    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755586    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mgbg\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.755802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.755638    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.756043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.756008    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.756204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.756180    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.756204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.756196    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.756294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.756215    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.756294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.756260    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.757820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.757800    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.757926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.757910    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.775081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.775054    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.775247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.775229    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mgbg\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:42.891239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:42.891157    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:43.205355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:43.205283    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"]
Oct 30 21:23:43.208877 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:23:43.208849    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33268e64_f853_4363_9bbf_191ff48e9205.slice/crio-6a33d1833fcabe2ec7ffa92a533eb816f00c7e7e388ab614dc12caf6d7e411fa WatchSource:0}: Error finding container 6a33d1833fcabe2ec7ffa92a533eb816f00c7e7e388ab614dc12caf6d7e411fa: Status 404 returned error can't find the container with id 6a33d1833fcabe2ec7ffa92a533eb816f00c7e7e388ab614dc12caf6d7e411fa
Oct 30 21:23:43.211043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:43.211014    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:43.211138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:43.211084    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:43.211138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:43.211127    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:44.167377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:44.167344    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" event={"ID":"33268e64-f853-4363-9bbf-191ff48e9205","Type":"ContainerStarted","Data":"7a60d8e66e897af2ad18b807c5298ce85fa953f5effce56fbf70a32a70b55e4b"}
Oct 30 21:23:44.167813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:44.167395    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" event={"ID":"33268e64-f853-4363-9bbf-191ff48e9205","Type":"ContainerStarted","Data":"6a33d1833fcabe2ec7ffa92a533eb816f00c7e7e388ab614dc12caf6d7e411fa"}
Oct 30 21:23:44.242622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:44.236481    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podStartSLOduration=2.236461221 podStartE2EDuration="2.236461221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:44.22379421 +0000 UTC m=+807.146252508" watchObservedRunningTime="2026-10-30 21:23:44.236461221 +0000 UTC m=+807.158919518"
Oct 30 21:23:44.892255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:44.892219    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:44.893673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:44.893651    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:23:44.893784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:44.893701    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:23:45.898623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:45.893992    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:23:45.898623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:45.894061    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:23:46.891860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:46.891823    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:23:46.892033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:46.891886    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:23:47.892431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:47.892395    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:23:47.892961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:47.892465    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:23:48.892301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:48.892260    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:23:48.892506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:48.892324    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 21:23:49.895897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:49.895867    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:49.896301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:49.896140    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:49.896928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:49.896909    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:23:53.835651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:53.834726    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:23:53.835651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:23:53.834824    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:01.074472 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:24:01.074393    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:24:08.970951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:08.970910    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"]
Oct 30 21:24:08.976359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:08.975689    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:08.998371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:08.997985    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"]
Oct 30 21:24:09.122802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.122765    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.122831    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.122940    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.123070    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.123104    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.123120    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.123147    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.123257    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.123334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.123316    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbbv\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224219    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224266    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224323    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224347    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224418    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224459    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224515    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224543    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbbv\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.224653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224619    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.225017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.225017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224852    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.225017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.224987    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.225277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.225249    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.225386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.225303    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.226841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.226815    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.227079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.227059    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.243347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.243317    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.243468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.243454    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbbv\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.315381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.315348    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:09.564724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.564692    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"]
Oct 30 21:24:09.568859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.568837    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.584305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.584272    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"]
Oct 30 21:24:09.589962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.589941    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.594556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.594531    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"]
Oct 30 21:24:09.612191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.607386    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"]
Oct 30 21:24:09.651508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.651477    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"]
Oct 30 21:24:09.654859 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:09.654831    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367c979a_ab4b_4145_af58_dfb2df6c1833.slice/crio-f93372b6166dbf10d006a1f8f0b943296b952603279b8b342bd0ba31927af238 WatchSource:0}: Error finding container f93372b6166dbf10d006a1f8f0b943296b952603279b8b342bd0ba31927af238: Status 404 returned error can't find the container with id f93372b6166dbf10d006a1f8f0b943296b952603279b8b342bd0ba31927af238
Oct 30 21:24:09.657004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.656973    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.657088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.657038    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.657088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.657068    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.731495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731463    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.731651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731574    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.731726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731675    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556gq\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.731726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731710    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.731852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731836    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.731918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731863    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.731980    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732015    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.732132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732109    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732146    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.732230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732212    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.732281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732266    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732294    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.732472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732343    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732413    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl779\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732478    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.732589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732509    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.732589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.732546    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.834244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834154    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.834244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834210    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.834244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834242    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.834533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834270    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl779\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.834533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834315    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.834840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834616    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.834840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834665    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.834840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834745    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.834840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834754    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.834840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834794    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834843    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556gq\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834870    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834970    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.834997    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835051    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835081    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835118    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835134    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835144    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835141    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835204    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835458    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.835704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835523    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.835873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.835758    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.836313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.836283    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.836665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.836639    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.836818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.836699    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.837109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.836490    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.837533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.837508    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.838108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.838087    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.838509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.838486    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.839562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.839541    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.853327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.853296    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.853435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.853331    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl779\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.853975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.853953    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.854051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.853978    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556gq\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:09.895676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.895632    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:09.910057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:09.910029    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:10.226319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.226210    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"]
Oct 30 21:24:10.229128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.229105    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"]
Oct 30 21:24:10.231060 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:10.231031    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac9bb90_1551_4066_ad6c_df13b6e99042.slice/crio-2736446f3ad4b436f6f26f9f78a6bc7301a2d034a85e38a6b50fb9f21ae6ee3f WatchSource:0}: Error finding container 2736446f3ad4b436f6f26f9f78a6bc7301a2d034a85e38a6b50fb9f21ae6ee3f: Status 404 returned error can't find the container with id 2736446f3ad4b436f6f26f9f78a6bc7301a2d034a85e38a6b50fb9f21ae6ee3f
Oct 30 21:24:10.233917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.233878    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.234013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.233966    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.234013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.234007    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.235682 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:10.235660    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afb3395_41cd_4c84_89cb_22e23b9d08d3.slice/crio-6ef13f3abdb8434a798e16cbb5a5b4aa08a39d397901c7c513b87b7faedc2ae2 WatchSource:0}: Error finding container 6ef13f3abdb8434a798e16cbb5a5b4aa08a39d397901c7c513b87b7faedc2ae2: Status 404 returned error can't find the container with id 6ef13f3abdb8434a798e16cbb5a5b4aa08a39d397901c7c513b87b7faedc2ae2
Oct 30 21:24:10.242500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.242468    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.242577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.242531    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.242633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.242583    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.458632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.457700    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" event={"ID":"0afb3395-41cd-4c84-89cb-22e23b9d08d3","Type":"ContainerStarted","Data":"d266ef50fac4bd2b4f5ccc7f321425cde5d8fa04457b482a6f3b423a83fdd00e"}
Oct 30 21:24:10.458632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.457768    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" event={"ID":"0afb3395-41cd-4c84-89cb-22e23b9d08d3","Type":"ContainerStarted","Data":"6ef13f3abdb8434a798e16cbb5a5b4aa08a39d397901c7c513b87b7faedc2ae2"}
Oct 30 21:24:10.460913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.460875    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" event={"ID":"cac9bb90-1551-4066-ad6c-df13b6e99042","Type":"ContainerStarted","Data":"7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da"}
Oct 30 21:24:10.461079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.460946    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" event={"ID":"cac9bb90-1551-4066-ad6c-df13b6e99042","Type":"ContainerStarted","Data":"2736446f3ad4b436f6f26f9f78a6bc7301a2d034a85e38a6b50fb9f21ae6ee3f"}
Oct 30 21:24:10.463018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.462986    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" event={"ID":"367c979a-ab4b-4145-af58-dfb2df6c1833","Type":"ContainerStarted","Data":"158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6"}
Oct 30 21:24:10.463204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.463184    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" event={"ID":"367c979a-ab4b-4145-af58-dfb2df6c1833","Type":"ContainerStarted","Data":"f93372b6166dbf10d006a1f8f0b943296b952603279b8b342bd0ba31927af238"}
Oct 30 21:24:10.550628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.546825    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podStartSLOduration=1.546806055 podStartE2EDuration="1.546806055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.499580656 +0000 UTC m=+833.422038953" watchObservedRunningTime="2026-10-30 21:24:10.546806055 +0000 UTC m=+833.469264352"
Oct 30 21:24:10.589511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.589445    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podStartSLOduration=2.589427261 podStartE2EDuration="2.589427261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.571807959 +0000 UTC m=+833.494266290" watchObservedRunningTime="2026-10-30 21:24:10.589427261 +0000 UTC m=+833.511885558"
Oct 30 21:24:10.639631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.638751    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podStartSLOduration=1.638729937 podStartE2EDuration="1.638729937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.629987846 +0000 UTC m=+833.552446144" watchObservedRunningTime="2026-10-30 21:24:10.638729937 +0000 UTC m=+833.561188234"
Oct 30 21:24:10.895814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.895695    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:10.910318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:10.910272    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:11.333638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.330970    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:11.337638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.334716    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:24:11.337638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.334768    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 21:24:11.902083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.901696    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:24:11.902083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.901758    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:24:11.922809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.922762    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:24:11.923097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:11.923063    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 21:24:12.324628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:12.316015    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:24:12.324628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:12.316075    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 21:24:12.899982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:12.899945    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:24:12.900563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:12.900533    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:24:12.914008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:12.913974    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:24:12.914250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:12.914225    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 21:24:13.331629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:13.323729    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:24:13.331629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:13.323823    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 21:24:13.896414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:13.896376    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:24:13.896594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:13.896428    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:24:13.912995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:13.912961    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:24:13.913537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:13.913508    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 21:24:14.326629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:14.318999    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:24:14.326629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:14.319061    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 21:24:14.896064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:14.896025    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:24:14.896341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:14.896310    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:24:14.914463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:14.914434    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:24:14.915017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:14.914987    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 21:24:15.334628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:15.332498    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:24:15.334628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:15.332562    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 21:24:15.902063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:15.901048    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:24:15.902063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:15.901111    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:24:15.913959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:15.913925    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:24:15.914224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:15.914196    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 21:24:16.317185 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:16.317147    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:24:16.317783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:16.317754    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 21:24:16.896290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:16.896244    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:24:16.896502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:16.896322    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 21:24:16.911224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:16.911188    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:24:16.911401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:16.911243    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 21:24:17.822499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.822431    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:17.823861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.823841    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:17.824674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.824652    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:24:17.901087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.901059    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:17.901372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.901347    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:17.902291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.902271    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:24:17.916219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.916195    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:17.916545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.916516    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:17.917510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:17.917493    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:24:31.072652 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:24:31.072592    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:24:47.762305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.762268    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"]
Oct 30 21:24:47.767522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.767500    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.783651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.783591    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"]
Oct 30 21:24:47.792159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.792131    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:24:47.792159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.792138    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:24:47.792370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.792197    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:24:47.792370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.792207    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d93cf183177\""
Oct 30 21:24:47.792370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.792227    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:24:47.795331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.795309    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dmwj5\""
Oct 30 21:24:47.838389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.838358    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.838578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.838414    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.838578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.838547    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.838755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.838668    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkr8\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.939579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.939545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.939799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.939594    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.939799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.939665    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.939799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.939725    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkr8\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.940466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.940439    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.940576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.940471    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.942625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.942586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:47.957994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:47.957969    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkr8\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8\") pod \"payload-processing-external-model-d4877f77-ctxcz\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:48.081856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:48.081824    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:48.382296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:48.382222    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"]
Oct 30 21:24:48.385754 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:48.385729    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aab010_a29b_45c5_9e3d_ae5bc1cf9e17.slice/crio-444fe0972f4e813bae7666ec78739ac81d836b66aa2b93f860b12ec782ab9785 WatchSource:0}: Error finding container 444fe0972f4e813bae7666ec78739ac81d836b66aa2b93f860b12ec782ab9785: Status 404 returned error can't find the container with id 444fe0972f4e813bae7666ec78739ac81d836b66aa2b93f860b12ec782ab9785
Oct 30 21:24:48.859668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:48.859633    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" event={"ID":"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17","Type":"ContainerStarted","Data":"d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b"}
Oct 30 21:24:48.860111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:48.859697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" event={"ID":"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17","Type":"ContainerStarted","Data":"444fe0972f4e813bae7666ec78739ac81d836b66aa2b93f860b12ec782ab9785"}
Oct 30 21:24:48.860111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:48.859725    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:24:48.900801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:48.900752    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" podStartSLOduration=1.900738117 podStartE2EDuration="1.900738117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:48.897659813 +0000 UTC m=+871.820118111" watchObservedRunningTime="2026-10-30 21:24:48.900738117 +0000 UTC m=+871.823196413"
Oct 30 21:24:50.821682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.821646    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"]
Oct 30 21:24:50.825815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.825797    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:50.841785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.841757    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"]
Oct 30 21:24:50.850315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.850265    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:50.850315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.850273    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:50.850508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.850346    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:50.851871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.851850    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:24:50.974458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.974416    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:50.974686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.974471    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:50.974686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:50.974664    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stf2\" (UniqueName: \"kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.075658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.075546    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.075658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.075592    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.075900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.075683    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stf2\" (UniqueName: \"kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.076226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.076199    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.078024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.078006    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.095403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.095375    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stf2\" (UniqueName: \"kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2\") pod \"maas-api-revoke-sub-41b43f0d-nlkqq\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.139173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.139146    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:51.439976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.439905    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"]
Oct 30 21:24:51.446020 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:51.445989    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482eeced_b283_4352_b36e_fdbc55e94cb5.slice/crio-14276afccffdbae65ec7224e27f57549a7b1029d4c7a0bc4de81c50baccf5793 WatchSource:0}: Error finding container 14276afccffdbae65ec7224e27f57549a7b1029d4c7a0bc4de81c50baccf5793: Status 404 returned error can't find the container with id 14276afccffdbae65ec7224e27f57549a7b1029d4c7a0bc4de81c50baccf5793
Oct 30 21:24:51.881271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.881238    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq" event={"ID":"482eeced-b283-4352-b36e-fdbc55e94cb5","Type":"ContainerStarted","Data":"b1e4899ef26d2942e06831b28534bb9616486a06852835c2d9bbd7445c31494a"}
Oct 30 21:24:51.881724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.881294    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq" event={"ID":"482eeced-b283-4352-b36e-fdbc55e94cb5","Type":"ContainerStarted","Data":"14276afccffdbae65ec7224e27f57549a7b1029d4c7a0bc4de81c50baccf5793"}
Oct 30 21:24:51.920266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:51.920220    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq" podStartSLOduration=1.920207251 podStartE2EDuration="1.920207251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:51.919166172 +0000 UTC m=+874.841624469" watchObservedRunningTime="2026-10-30 21:24:51.920207251 +0000 UTC m=+874.842665548"
Oct 30 21:24:52.130187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.130152    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"]
Oct 30 21:24:52.135074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.135043    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.149475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.149447    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"distinct-w0-49738b-kserve-self-signed-certs\""
Oct 30 21:24:52.152506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.152478    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"]
Oct 30 21:24:52.291962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.291931    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.292133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.292040    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.292291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.292274    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.292546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.292527    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.292642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.292583    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxv8j\" (UniqueName: \"kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.292787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.292767    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.397797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.397652    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.397797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.397724    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.397797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.397771    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.398074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.397847    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.398074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.397987    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.398074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.398044    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxv8j\" (UniqueName: \"kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.400724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.399902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.400724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.400142    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.400724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.400351    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.418488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.418450    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.421466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.421438    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.423107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.423080    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxv8j\" (UniqueName: \"kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j\") pod \"distinct-w0-49738b-kserve-56d4667b9d-zv7wq\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") " pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.454161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.454113    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:52.761454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.761371    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"]
Oct 30 21:24:52.764668 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:52.764638    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75b95c9e_4fd2_402b_ad68_0025760c7394.slice/crio-dd5f0c0bb23b6da563c2e981bd485c8715b7f147471891a1be8ae9359b261d99 WatchSource:0}: Error finding container dd5f0c0bb23b6da563c2e981bd485c8715b7f147471891a1be8ae9359b261d99: Status 404 returned error can't find the container with id dd5f0c0bb23b6da563c2e981bd485c8715b7f147471891a1be8ae9359b261d99
Oct 30 21:24:52.896954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.896919    6695 generic.go:356] "Generic (PLEG): container finished" podID="482eeced-b283-4352-b36e-fdbc55e94cb5" containerID="b1e4899ef26d2942e06831b28534bb9616486a06852835c2d9bbd7445c31494a" exitCode=0
Oct 30 21:24:52.897351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.897000    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq" event={"ID":"482eeced-b283-4352-b36e-fdbc55e94cb5","Type":"ContainerDied","Data":"b1e4899ef26d2942e06831b28534bb9616486a06852835c2d9bbd7445c31494a"}
Oct 30 21:24:52.898792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.898766    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" event={"ID":"75b95c9e-4fd2-402b-ad68-0025760c7394","Type":"ContainerStarted","Data":"f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c"}
Oct 30 21:24:52.898898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.898798    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" event={"ID":"75b95c9e-4fd2-402b-ad68-0025760c7394","Type":"ContainerStarted","Data":"dd5f0c0bb23b6da563c2e981bd485c8715b7f147471891a1be8ae9359b261d99"}
Oct 30 21:24:52.974872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:52.974806    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podStartSLOduration=0.974788399 podStartE2EDuration="974.788399ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:52.971584197 +0000 UTC m=+875.894042496" watchObservedRunningTime="2026-10-30 21:24:52.974788399 +0000 UTC m=+875.897246697"
Oct 30 21:24:53.455188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:53.455159    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:24:54.072503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.072442    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:54.244665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.244530    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca\") pod \"482eeced-b283-4352-b36e-fdbc55e94cb5\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") "
Oct 30 21:24:54.244665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.244640    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2\" (UniqueName: \"kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2\") pod \"482eeced-b283-4352-b36e-fdbc55e94cb5\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") "
Oct 30 21:24:54.244995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.244756    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca\") pod \"482eeced-b283-4352-b36e-fdbc55e94cb5\" (UID: \"482eeced-b283-4352-b36e-fdbc55e94cb5\") "
Oct 30 21:24:54.245990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.245952    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca" pod "482eeced-b283-4352-b36e-fdbc55e94cb5" (UID: "482eeced-b283-4352-b36e-fdbc55e94cb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:54.248506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.247805    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2" pod "482eeced-b283-4352-b36e-fdbc55e94cb5" (UID: "482eeced-b283-4352-b36e-fdbc55e94cb5"). InnerVolumeSpecName "kube-api-access-5stf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:54.250755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.250724    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca" pod "482eeced-b283-4352-b36e-fdbc55e94cb5" (UID: "482eeced-b283-4352-b36e-fdbc55e94cb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:54.347179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.347151    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:24:54.347179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.347177    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5stf2\" (UniqueName: \"kubernetes.io/projected/482eeced-b283-4352-b36e-fdbc55e94cb5-kube-api-access-5stf2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:24:54.347179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.347187    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482eeced-b283-4352-b36e-fdbc55e94cb5-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:24:54.748627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.747512    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"]
Oct 30 21:24:54.859486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.857049    6695 cpu_manager.go:414] "containerMap: removing container" podUID="482eeced-b283-4352-b36e-fdbc55e94cb5" containerName="revoke-keys"
Oct 30 21:24:54.859486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.857094    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482eeced-b283-4352-b36e-fdbc55e94cb5" containerName="revoke-keys"
Oct 30 21:24:54.859486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.857289    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="482eeced-b283-4352-b36e-fdbc55e94cb5" containerName="revoke-keys"
Oct 30 21:24:54.921628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.876645    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"]
Oct 30 21:24:54.921628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.876770    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957357    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957440    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957462    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6549\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957510    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957582    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957641    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957722    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957789    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:54.980685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:54.957815    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.016682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.001922    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"
Oct 30 21:24:55.016682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.008339    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq" event={"ID":"482eeced-b283-4352-b36e-fdbc55e94cb5","Type":"ContainerDied","Data":"14276afccffdbae65ec7224e27f57549a7b1029d4c7a0bc4de81c50baccf5793"}
Oct 30 21:24:55.016682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.008394    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14276afccffdbae65ec7224e27f57549a7b1029d4c7a0bc4de81c50baccf5793"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.061357    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.062381    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063361    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063404    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063495    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063572    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063597    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063666    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063743    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063769    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6549\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.063829    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.064626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.064206    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.068635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.064768    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.068635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.065573    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.078325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.078297    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.090337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.090308    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.093316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.093288    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6549\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.094861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.094821    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:55.298628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:55.297872    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:56.617628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:56.617266    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"]
Oct 30 21:24:56.667098 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:24:56.667063    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e3522c_639c_4905_97be_e55c59e193f7.slice/crio-1474153cd46a1dda800e06eb48c603399c1ba9347298d79bd59a4d34552bf74d WatchSource:0}: Error finding container 1474153cd46a1dda800e06eb48c603399c1ba9347298d79bd59a4d34552bf74d: Status 404 returned error can't find the container with id 1474153cd46a1dda800e06eb48c603399c1ba9347298d79bd59a4d34552bf74d
Oct 30 21:24:56.736658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:56.729927    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:56.737123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:56.737085    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:56.737327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:56.737305    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:56.746798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:56.746760    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:56.878160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:56.876704    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.024064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:57.016989    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.350804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:57.350756    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.788668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:57.774590    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.894628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:57.870201    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:24:57.982055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:57.982009    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:24:57.982960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:57.982935    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:24:58.029040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.026726    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:58.029040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.027416    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:24:58.330637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.330058    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" event={"ID":"90e3522c-639c-4905-97be-e55c59e193f7","Type":"ContainerStarted","Data":"1474153cd46a1dda800e06eb48c603399c1ba9347298d79bd59a4d34552bf74d"}
Oct 30 21:24:58.437638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.437418    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podStartSLOduration=4.437397401 podStartE2EDuration="4.437397401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:58.408908101 +0000 UTC m=+881.331366401" watchObservedRunningTime="2026-10-30 21:24:58.437397401 +0000 UTC m=+881.359855700"
Oct 30 21:24:58.832402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.832371    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:24:58.982853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.868682    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:24:58.998289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.910231    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:24:59.024036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:58.982755    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.024437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.005986    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:24:59.038223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.029705    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.098758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.023980    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.390687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.326375    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:24:59.390687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.367504    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:24:59.390687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.367553    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:24:59.473496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.473463    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:24:59.473800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.473781    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:24:59.498520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.498490    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" event={"ID":"90e3522c-639c-4905-97be-e55c59e193f7","Type":"ContainerStarted","Data":"f3c65fc847382165bc11b8a0777312557e36afcc8372e4c444216b74528d0bba"}
Oct 30 21:24:59.857752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.857702    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.937984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:24:59.936812    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:25:00.103764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:00.103716    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:00.324775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:00.324736    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:00.350741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:00.350685    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:00.862649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:00.861704    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:00.862649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:00.862446    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:25:00.910626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:00.905982    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:25:01.308401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:01.308362    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:01.308678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:01.308650    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:01.531736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:01.528795    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:25:01.693757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:01.676664    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:01.744690 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:25:01.690498    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:25:02.300325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:02.300284    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:02.300845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:02.300351    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:02.460543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:02.459045    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:02.480276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:02.480243    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:02.748586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:02.748501    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:03.299885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:03.299835    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:03.300065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:03.299904    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:04.299786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:04.299751    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:04.300189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:04.299809    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:05.299383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:05.299346    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:05.299566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:05.299400    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:06.303844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:06.303809    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:25:06.304665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:06.304622    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 21:25:07.355885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:07.355851    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:25:07.356713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:07.356684    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:25:07.357497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:07.357477    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:25:16.786649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.786597    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"]
Oct 30 21:25:16.789465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.789446    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:16.808089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.808058    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"]
Oct 30 21:25:16.816225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.816197    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:16.816410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.816246    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:16.816410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.816281    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:16.816410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.816385    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:25:16.897288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.897252    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5nm\" (UniqueName: \"kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:16.897476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.897416    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:16.897528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:16.897479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.007324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.007271    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5nm\" (UniqueName: \"kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.007503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.007490    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.007697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.007576    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.009641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.009350    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.029086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.029050    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.043150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.043072    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5nm\" (UniqueName: \"kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm\") pod \"maas-api-revoke-sub-df70818f-l82wn\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.107548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.107507    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:17.319846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.319770    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:25:17.328297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.328270    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:25:17.408858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.408776    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"]
Oct 30 21:25:17.411783 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:17.411749    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f40ab1_058d_49d6_8098_6b92e5421aa2.slice/crio-f4faa6cdf2e7088c85c754811e1f1abd12f65e7ab7d67bc42afaad8f7f52f68d WatchSource:0}: Error finding container f4faa6cdf2e7088c85c754811e1f1abd12f65e7ab7d67bc42afaad8f7f52f68d: Status 404 returned error can't find the container with id f4faa6cdf2e7088c85c754811e1f1abd12f65e7ab7d67bc42afaad8f7f52f68d
Oct 30 21:25:17.911744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.911628    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn" event={"ID":"74f40ab1-058d-49d6-8098-6b92e5421aa2","Type":"ContainerStarted","Data":"ea5f94b14d3d4ee3b9771180bdee312bd6d0f790158d1b5392b94eab7e50327e"}
Oct 30 21:25:17.911744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.911694    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn" event={"ID":"74f40ab1-058d-49d6-8098-6b92e5421aa2","Type":"ContainerStarted","Data":"f4faa6cdf2e7088c85c754811e1f1abd12f65e7ab7d67bc42afaad8f7f52f68d"}
Oct 30 21:25:17.953946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:17.953875    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn" podStartSLOduration=1.953848066 podStartE2EDuration="1.953848066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:17.948405475 +0000 UTC m=+900.870863773" watchObservedRunningTime="2026-10-30 21:25:17.953848066 +0000 UTC m=+900.876306365"
Oct 30 21:25:18.919547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:18.919511    6695 generic.go:356] "Generic (PLEG): container finished" podID="74f40ab1-058d-49d6-8098-6b92e5421aa2" containerID="ea5f94b14d3d4ee3b9771180bdee312bd6d0f790158d1b5392b94eab7e50327e" exitCode=0
Oct 30 21:25:18.920050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:18.919556    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn" event={"ID":"74f40ab1-058d-49d6-8098-6b92e5421aa2","Type":"ContainerDied","Data":"ea5f94b14d3d4ee3b9771180bdee312bd6d0f790158d1b5392b94eab7e50327e"}
Oct 30 21:25:20.076793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.076768    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:20.152901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.152867    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm\" (UniqueName: \"kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm\") pod \"74f40ab1-058d-49d6-8098-6b92e5421aa2\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") "
Oct 30 21:25:20.153098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.152984    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca\") pod \"74f40ab1-058d-49d6-8098-6b92e5421aa2\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") "
Oct 30 21:25:20.153098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.153064    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca\") pod \"74f40ab1-058d-49d6-8098-6b92e5421aa2\" (UID: \"74f40ab1-058d-49d6-8098-6b92e5421aa2\") "
Oct 30 21:25:20.153434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.153401    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca" pod "74f40ab1-058d-49d6-8098-6b92e5421aa2" (UID: "74f40ab1-058d-49d6-8098-6b92e5421aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:20.153762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.153738    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:20.155371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.155337    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca" pod "74f40ab1-058d-49d6-8098-6b92e5421aa2" (UID: "74f40ab1-058d-49d6-8098-6b92e5421aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:20.155484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.155440    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm" pod "74f40ab1-058d-49d6-8098-6b92e5421aa2" (UID: "74f40ab1-058d-49d6-8098-6b92e5421aa2"). InnerVolumeSpecName "kube-api-access-bn5nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:20.255022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.254945    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f40ab1-058d-49d6-8098-6b92e5421aa2-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:20.255022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.254974    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5nm\" (UniqueName: \"kubernetes.io/projected/74f40ab1-058d-49d6-8098-6b92e5421aa2-kube-api-access-bn5nm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:20.933395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.933361    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn" event={"ID":"74f40ab1-058d-49d6-8098-6b92e5421aa2","Type":"ContainerDied","Data":"f4faa6cdf2e7088c85c754811e1f1abd12f65e7ab7d67bc42afaad8f7f52f68d"}
Oct 30 21:25:20.933565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.933400    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"
Oct 30 21:25:20.933565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:20.933408    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4faa6cdf2e7088c85c754811e1f1abd12f65e7ab7d67bc42afaad8f7f52f68d"
Oct 30 21:25:21.947413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.947377    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"]
Oct 30 21:25:21.947973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.947958    6695 cpu_manager.go:414] "containerMap: removing container" podUID="74f40ab1-058d-49d6-8098-6b92e5421aa2" containerName="revoke-keys"
Oct 30 21:25:21.948030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.947974    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f40ab1-058d-49d6-8098-6b92e5421aa2" containerName="revoke-keys"
Oct 30 21:25:21.948084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.948075    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f40ab1-058d-49d6-8098-6b92e5421aa2" containerName="revoke-keys"
Oct 30 21:25:21.951553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.951538    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:21.967927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.967902    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"]
Oct 30 21:25:21.976010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.975988    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:22.003462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.992295    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:22.003462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.992405    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:22.003462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.993774    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.003462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.993908    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.003462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.994011    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wx94\" (UniqueName: \"kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.003462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:21.994381    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:25:22.094879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.094841    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.095068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.094996    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.095133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.095102    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wx94\" (UniqueName: \"kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.095884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.095858    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.097425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.097402    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.113778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.113750    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wx94\" (UniqueName: \"kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94\") pod \"maas-api-revoke-sub-89153b7e-q6c56\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.265750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.265664    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:22.388560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.388527    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"]
Oct 30 21:25:22.565888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.565854    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"]
Oct 30 21:25:22.569542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:22.569513    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2bae61_33c4_4d29_80b9_0fab06281e84.slice/crio-1f9b0badc78b1b760a77933228fae2f056ac1f8cbd5eeeae96df52967984e3d2 WatchSource:0}: Error finding container 1f9b0badc78b1b760a77933228fae2f056ac1f8cbd5eeeae96df52967984e3d2: Status 404 returned error can't find the container with id 1f9b0badc78b1b760a77933228fae2f056ac1f8cbd5eeeae96df52967984e3d2
Oct 30 21:25:22.948414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.948373    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56" event={"ID":"7d2bae61-33c4-4d29-80b9-0fab06281e84","Type":"ContainerStarted","Data":"7b0e73ecd1387443fa1aa8b50c74cc97754ba38f32b6322ce3c88f163929856f"}
Oct 30 21:25:22.948821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.948438    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56" event={"ID":"7d2bae61-33c4-4d29-80b9-0fab06281e84","Type":"ContainerStarted","Data":"1f9b0badc78b1b760a77933228fae2f056ac1f8cbd5eeeae96df52967984e3d2"}
Oct 30 21:25:22.998293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:22.998237    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56" podStartSLOduration=1.9982222969999999 podStartE2EDuration="1.998222297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:22.984986899 +0000 UTC m=+905.907445199" watchObservedRunningTime="2026-10-30 21:25:22.998222297 +0000 UTC m=+905.920680594"
Oct 30 21:25:23.956477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:23.956444    6695 generic.go:356] "Generic (PLEG): container finished" podID="7d2bae61-33c4-4d29-80b9-0fab06281e84" containerID="7b0e73ecd1387443fa1aa8b50c74cc97754ba38f32b6322ce3c88f163929856f" exitCode=0
Oct 30 21:25:23.956913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:23.956538    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56" event={"ID":"7d2bae61-33c4-4d29-80b9-0fab06281e84","Type":"ContainerDied","Data":"7b0e73ecd1387443fa1aa8b50c74cc97754ba38f32b6322ce3c88f163929856f"}
Oct 30 21:25:25.202154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.202111    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"]
Oct 30 21:25:25.206669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.206635    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.242159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.242125    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"]
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251216    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251269    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251310    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251349    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251391    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251424    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251474    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251572    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.252144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.251649    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpv52\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.265466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.264715    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"]
Oct 30 21:25:25.271644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.270935    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.284779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.284743    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:25.289557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.289518    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"]
Oct 30 21:25:25.354247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354207    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca\") pod \"7d2bae61-33c4-4d29-80b9-0fab06281e84\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") "
Oct 30 21:25:25.354247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354259    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca\") pod \"7d2bae61-33c4-4d29-80b9-0fab06281e84\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") "
Oct 30 21:25:25.354664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354404    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94\" (UniqueName: \"kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94\") pod \"7d2bae61-33c4-4d29-80b9-0fab06281e84\" (UID: \"7d2bae61-33c4-4d29-80b9-0fab06281e84\") "
Oct 30 21:25:25.354864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354838    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.354977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354883    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.354977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354929    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.354977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354963    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.354988    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355031    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355070    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355104    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355140    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355189    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355224    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355277    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355366    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wns\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355414    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355462    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355500    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355530    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpv52\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.355638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.355558    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.356458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.356430    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca" pod "7d2bae61-33c4-4d29-80b9-0fab06281e84" (UID: "7d2bae61-33c4-4d29-80b9-0fab06281e84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:25.357424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.357400    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.357735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.357648    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.358418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.358002    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.358418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.358051    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.358939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.358913    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.359535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.359512    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca" pod "7d2bae61-33c4-4d29-80b9-0fab06281e84" (UID: "7d2bae61-33c4-4d29-80b9-0fab06281e84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:25.359697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.359589    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94" pod "7d2bae61-33c4-4d29-80b9-0fab06281e84" (UID: "7d2bae61-33c4-4d29-80b9-0fab06281e84"). InnerVolumeSpecName "kube-api-access-7wx94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:25.360463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.360437    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.361258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.361236    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.384432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.384396    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.385835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.385806    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpv52\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.458920    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459043    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459080    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459262    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wns\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459327    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459373    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459429    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459547    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459574    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459731    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459750    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bae61-33c4-4d29-80b9-0fab06281e84-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:25.460637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.459768    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wx94\" (UniqueName: \"kubernetes.io/projected/7d2bae61-33c4-4d29-80b9-0fab06281e84-kube-api-access-7wx94\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:25.474639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.473398    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.499628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.497742    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.499628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.498170    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.499628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.499574    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.499935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.499855    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.518746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.513509    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.527550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.527474    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.532258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.532185    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.541632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.541229    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wns\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:25.577310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.576899    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:25.597441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:25.597220    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:26.334695 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:26.334072    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722b7290_7400_4077_89a8_c30112abc9cb.slice/crio-d6e9e8e0e2684a26b142844b7d7b15ff5a0402289ad381907aea4c35ab7edc8d WatchSource:0}: Error finding container d6e9e8e0e2684a26b142844b7d7b15ff5a0402289ad381907aea4c35ab7edc8d: Status 404 returned error can't find the container with id d6e9e8e0e2684a26b142844b7d7b15ff5a0402289ad381907aea4c35ab7edc8d
Oct 30 21:25:26.382634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.381511    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.382634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.381640    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.382634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.381705    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.494982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.494887    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"]
Oct 30 21:25:26.544859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.544812    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56" event={"ID":"7d2bae61-33c4-4d29-80b9-0fab06281e84","Type":"ContainerDied","Data":"1f9b0badc78b1b760a77933228fae2f056ac1f8cbd5eeeae96df52967984e3d2"}
Oct 30 21:25:26.545090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.545072    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f9b0badc78b1b760a77933228fae2f056ac1f8cbd5eeeae96df52967984e3d2"
Oct 30 21:25:26.545252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.545238    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"
Oct 30 21:25:26.722742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.722684    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:26.757618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.757527    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"]
Oct 30 21:25:26.796204 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:26.796158    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404cec42_23cb_4e3c_ae29_fb05c11f1da7.slice/crio-f8926475c250e7440672bc056ad69efbb4fb38e0ad7a5016447b98446b74e968 WatchSource:0}: Error finding container f8926475c250e7440672bc056ad69efbb4fb38e0ad7a5016447b98446b74e968: Status 404 returned error can't find the container with id f8926475c250e7440672bc056ad69efbb4fb38e0ad7a5016447b98446b74e968
Oct 30 21:25:26.808738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.808697    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.809042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.809005    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.809237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:26.809201    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:27.403696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.402872    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" podUID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" containerName="payload-processing" containerID="cri-o://d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b" gracePeriod=25
Oct 30 21:25:27.686204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.686111    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" event={"ID":"722b7290-7400-4077-89a8-c30112abc9cb","Type":"ContainerStarted","Data":"c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240"}
Oct 30 21:25:27.686468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.686441    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" event={"ID":"722b7290-7400-4077-89a8-c30112abc9cb","Type":"ContainerStarted","Data":"d6e9e8e0e2684a26b142844b7d7b15ff5a0402289ad381907aea4c35ab7edc8d"}
Oct 30 21:25:27.777704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.777033    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" event={"ID":"404cec42-23cb-4e3c-ae29-fb05c11f1da7","Type":"ContainerStarted","Data":"57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c"}
Oct 30 21:25:27.777704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.777101    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" event={"ID":"404cec42-23cb-4e3c-ae29-fb05c11f1da7","Type":"ContainerStarted","Data":"f8926475c250e7440672bc056ad69efbb4fb38e0ad7a5016447b98446b74e968"}
Oct 30 21:25:27.801623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.791422    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podStartSLOduration=2.79139825 podStartE2EDuration="2.79139825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:27.774672829 +0000 UTC m=+910.697131151" watchObservedRunningTime="2026-10-30 21:25:27.79139825 +0000 UTC m=+910.713856549"
Oct 30 21:25:27.834620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:27.834534    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podStartSLOduration=2.834512022 podStartE2EDuration="2.834512022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:27.829081761 +0000 UTC m=+910.751540084" watchObservedRunningTime="2026-10-30 21:25:27.834512022 +0000 UTC m=+910.756970344"
Oct 30 21:25:28.584674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:28.581035    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:28.584674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:28.581184    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:28.584674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:28.581230    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:28.606634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:28.605308    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:28.613632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:28.610597    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:28.613632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:28.610675    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:29.335447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.335397    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:29.373667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.370693    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:29.580734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.580698    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:29.581036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.581002    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:29.603990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.603897    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:29.603990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.603956    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:29.623627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.613237    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:29.767640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.731943    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:29.917224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:29.917127    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" podUID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:30.034628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.033752    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:30.340637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.338729    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:30.340637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.339561    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:25:30.371182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.371141    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:30.371707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.371680    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:25:30.670984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.667743    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:30.670984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.668209    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:25:30.737124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.736229    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:30.737124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.736299    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:30.760873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.747207    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:30.760873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.747260    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:30.840645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.830378    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:30.840645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:30.830792    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:31.004943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.004744    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:31.004943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.004809    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:25:31.086341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.086319    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:25:31.090197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.090032    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:25:31.125838 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:25:31.125793    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:25:31.470701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.470666    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:25:31.498382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498353    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials\") pod \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") "
Oct 30 21:25:31.498552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498396    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume\") pod \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") "
Oct 30 21:25:31.498552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498434    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume\") pod \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") "
Oct 30 21:25:31.498678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498562    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8\") pod \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\" (UID: \"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17\") "
Oct 30 21:25:31.498894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498864    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume" pod "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" (UID: "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:31.498983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498919    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume" pod "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" (UID: "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:31.499022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.498988    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:31.500799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.500771    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials" pod "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" (UID: "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:31.500799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.500785    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8" pod "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" (UID: "85aab010-a29b-45c5-9e3d-ae5bc1cf9e17"). InnerVolumeSpecName "kube-api-access-8gkr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:31.578430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.578394    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:31.578710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.578685    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:31.598396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.598370    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:31.598808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.598766    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:31.599929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.599905    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkr8\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-kube-api-access-8gkr8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:31.600074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.600061    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:31.600157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.600146    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:31.877559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.877188    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"]
Oct 30 21:25:31.878069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.877927    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7d2bae61-33c4-4d29-80b9-0fab06281e84" containerName="revoke-keys"
Oct 30 21:25:31.878069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.877949    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2bae61-33c4-4d29-80b9-0fab06281e84" containerName="revoke-keys"
Oct 30 21:25:31.878069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.877998    6695 cpu_manager.go:414] "containerMap: removing container" podUID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" containerName="payload-processing"
Oct 30 21:25:31.878069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.878006    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" containerName="payload-processing"
Oct 30 21:25:31.878284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.878173    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" containerName="payload-processing"
Oct 30 21:25:31.878284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.878193    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2bae61-33c4-4d29-80b9-0fab06281e84" containerName="revoke-keys"
Oct 30 21:25:31.882546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.882521    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:31.898277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.898249    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"distinct-2-w4-ea40ee-kserve-self-signed-certs\""
Oct 30 21:25:31.898598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.898577    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"]
Oct 30 21:25:31.903194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.903170    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:31.903385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.903368    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:31.903476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.903465    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:31.903577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.903568    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:31.903734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.903712    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:31.903811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:31.903744    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfqv\" (UniqueName: \"kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.004959    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005083    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfqv\" (UniqueName: \"kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005135    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005453    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005737    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005793    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005814    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.005989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.005829    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.006130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.006075    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.008450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.008427    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.009258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.009220    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.109255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.101484    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfqv\" (UniqueName: \"kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv\") pod \"distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.137674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.134337    6695 generic.go:356] "Generic (PLEG): container finished" podID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" containerID="d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b" exitCode=0
Oct 30 21:25:32.137674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.134515    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" event={"ID":"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17","Type":"ContainerDied","Data":"d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b"}
Oct 30 21:25:32.137674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.134567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz" event={"ID":"85aab010-a29b-45c5-9e3d-ae5bc1cf9e17","Type":"ContainerDied","Data":"444fe0972f4e813bae7666ec78739ac81d836b66aa2b93f860b12ec782ab9785"}
Oct 30 21:25:32.137674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.134616    6695 scope.go:122] "RemoveContainer" containerID="d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b"
Oct 30 21:25:32.137674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.134787    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"
Oct 30 21:25:32.196977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.196943    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:32.393443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.393353    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"]
Oct 30 21:25:32.401497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.400006    6695 scope.go:122] "RemoveContainer" containerID="d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b"
Oct 30 21:25:32.401497 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:25:32.400474    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b\": container with ID starting with d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b not found: ID does not exist" containerID="d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b"
Oct 30 21:25:32.401497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.400516    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b"} err="failed to get container status \"d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b\": rpc error: code = NotFound desc = could not find container \"d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b\": container with ID starting with d56ba0162caf70d85c349aae007b91297e5045dab43caee9c6feb45e6e0ed97b not found: ID does not exist"
Oct 30 21:25:32.409834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.407140    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"]
Oct 30 21:25:32.409834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.407181    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-ctxcz"]
Oct 30 21:25:32.409834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.407207    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"]
Oct 30 21:25:32.409834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.407304    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.427028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.426994    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:25:32.435627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.427944    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:32.457949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.436806    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:32.458166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.436896    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:32.475627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.475239    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.475627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.475479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l86rg\" (UniqueName: \"kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.475627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.475537    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.578243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.578215    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:32.578471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.578440    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:32.584859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.584783    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l86rg\" (UniqueName: \"kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.585824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.585069    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.585824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.585293    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.585824 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:25:32.585746    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aab010_a29b_45c5_9e3d_ae5bc1cf9e17.slice/crio-444fe0972f4e813bae7666ec78739ac81d836b66aa2b93f860b12ec782ab9785\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aab010_a29b_45c5_9e3d_ae5bc1cf9e17.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:25:32.588518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.588497    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.593202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.593183    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.598101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.598028    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:32.598101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.598063    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:32.608573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.608551    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l86rg\" (UniqueName: \"kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg\") pod \"maas-api-revoke-sub-a3a87881-kz4x2\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.809746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.809711    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:32.860651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:32.846648    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"]
Oct 30 21:25:33.365633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.351347    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podStartSLOduration=2.351323849 podStartE2EDuration="2.351323849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:33.337356375 +0000 UTC m=+916.259814674" watchObservedRunningTime="2026-10-30 21:25:33.351323849 +0000 UTC m=+916.273782149"
Oct 30 21:25:33.369274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.369236    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85aab010-a29b-45c5-9e3d-ae5bc1cf9e17" path="/var/lib/kubelet/pods/85aab010-a29b-45c5-9e3d-ae5bc1cf9e17/volumes"
Oct 30 21:25:33.370069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.370049    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"]
Oct 30 21:25:33.370302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.370267    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" event={"ID":"48a89f7d-9471-4be3-b234-010eb43156b7","Type":"ContainerStarted","Data":"71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54"}
Oct 30 21:25:33.370440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.370423    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" event={"ID":"48a89f7d-9471-4be3-b234-010eb43156b7","Type":"ContainerStarted","Data":"e91fe22cab2b8e1e67624577427b3bc820fca5326aaeef0a62b721b861f3d143"}
Oct 30 21:25:33.585141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.577931    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:33.585141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.577993    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:33.598639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.598286    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:33.598639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:33.598333    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:34.198108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.198062    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:34.277411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.277368    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2" event={"ID":"6e9a333b-d0e9-4727-bb9d-37682cd189ae","Type":"ContainerStarted","Data":"33121375c90b6290098973f81eab5687ca76e9c3d2155b4e34145f5b2c854cfd"}
Oct 30 21:25:34.277643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.277434    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2" event={"ID":"6e9a333b-d0e9-4727-bb9d-37682cd189ae","Type":"ContainerStarted","Data":"f6b4492e212999fcaba060e1ca181c026fbc5bac2c78a11db5a0f45c5834245f"}
Oct 30 21:25:34.363091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.354206    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2" podStartSLOduration=2.35418131 podStartE2EDuration="2.35418131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:34.322135025 +0000 UTC m=+917.244593324" watchObservedRunningTime="2026-10-30 21:25:34.35418131 +0000 UTC m=+917.276639609"
Oct 30 21:25:34.578348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.578251    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:34.578348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.578316    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:34.597976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.597943    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:34.598226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:34.598195    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:35.287177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:35.287134    6695 generic.go:356] "Generic (PLEG): container finished" podID="6e9a333b-d0e9-4727-bb9d-37682cd189ae" containerID="33121375c90b6290098973f81eab5687ca76e9c3d2155b4e34145f5b2c854cfd" exitCode=0
Oct 30 21:25:35.287358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:35.287221    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2" event={"ID":"6e9a333b-d0e9-4727-bb9d-37682cd189ae","Type":"ContainerDied","Data":"33121375c90b6290098973f81eab5687ca76e9c3d2155b4e34145f5b2c854cfd"}
Oct 30 21:25:35.578348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:35.578254    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:25:35.578348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:35.578307    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 21:25:35.597941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:35.597909    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:25:35.598085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:35.597969    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 21:25:36.572485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.572392    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:36.581024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.580994    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"]
Oct 30 21:25:36.581854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.581830    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6e9a333b-d0e9-4727-bb9d-37682cd189ae" containerName="revoke-keys"
Oct 30 21:25:36.581854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.581850    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9a333b-d0e9-4727-bb9d-37682cd189ae" containerName="revoke-keys"
Oct 30 21:25:36.582046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.582011    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9a333b-d0e9-4727-bb9d-37682cd189ae" containerName="revoke-keys"
Oct 30 21:25:36.587256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.587222    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.603641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.601517    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"]
Oct 30 21:25:36.609932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.608019    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:36.609932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.609130    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:36.618384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.617537    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:25:36.628791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.628498    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:36.629870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.629803    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:36.633029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.632983    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:25:36.656107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.656066    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg\" (UniqueName: \"kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg\") pod \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") "
Oct 30 21:25:36.656290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.656269    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca\") pod \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") "
Oct 30 21:25:36.656398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.656381    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca\") pod \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\" (UID: \"6e9a333b-d0e9-4727-bb9d-37682cd189ae\") "
Oct 30 21:25:36.656724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.656695    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca" pod "6e9a333b-d0e9-4727-bb9d-37682cd189ae" (UID: "6e9a333b-d0e9-4727-bb9d-37682cd189ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:36.656893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.656721    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llbs\" (UniqueName: \"kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.657013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.656992    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.657164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.657146    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.657477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.657455    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:36.658903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.658876    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg" pod "6e9a333b-d0e9-4727-bb9d-37682cd189ae" (UID: "6e9a333b-d0e9-4727-bb9d-37682cd189ae"). InnerVolumeSpecName "kube-api-access-l86rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:36.659176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.659153    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca" pod "6e9a333b-d0e9-4727-bb9d-37682cd189ae" (UID: "6e9a333b-d0e9-4727-bb9d-37682cd189ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:36.758230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.758174    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llbs\" (UniqueName: \"kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.758396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.758298    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.758396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.758389    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.759428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.759402    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.760107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.760053    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9a333b-d0e9-4727-bb9d-37682cd189ae-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:36.760107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.760081    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l86rg\" (UniqueName: \"kubernetes.io/projected/6e9a333b-d0e9-4727-bb9d-37682cd189ae-kube-api-access-l86rg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:36.765768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.765436    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.782714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.782671    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llbs\" (UniqueName: \"kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs\") pod \"maas-api-revoke-sub-c9f8581f-wcnk6\" (UID: \"f6359677-6168-4783-9c33-494489293799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:36.905073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:36.904980    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:37.213647 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:37.213586    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6359677_6168_4783_9c33_494489293799.slice/crio-ced733a7437f21bafe8efaa4179c1e2bd1e5e705aa673975db0e105b3981b975 WatchSource:0}: Error finding container ced733a7437f21bafe8efaa4179c1e2bd1e5e705aa673975db0e105b3981b975: Status 404 returned error can't find the container with id ced733a7437f21bafe8efaa4179c1e2bd1e5e705aa673975db0e105b3981b975
Oct 30 21:25:37.216691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:37.216669    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"]
Oct 30 21:25:37.316688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:37.316645    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6" event={"ID":"f6359677-6168-4783-9c33-494489293799","Type":"ContainerStarted","Data":"ced733a7437f21bafe8efaa4179c1e2bd1e5e705aa673975db0e105b3981b975"}
Oct 30 21:25:37.318026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:37.317999    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2" event={"ID":"6e9a333b-d0e9-4727-bb9d-37682cd189ae","Type":"ContainerDied","Data":"f6b4492e212999fcaba060e1ca181c026fbc5bac2c78a11db5a0f45c5834245f"}
Oct 30 21:25:37.318140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:37.318035    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"
Oct 30 21:25:37.318140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:37.318043    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b4492e212999fcaba060e1ca181c026fbc5bac2c78a11db5a0f45c5834245f"
Oct 30 21:25:38.325994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:38.325895    6695 generic.go:356] "Generic (PLEG): container finished" podID="f6359677-6168-4783-9c33-494489293799" containerID="69a96583497f6684f2dc16df0c3cd9d0e57414ba6443b6c1ef6e097b223f6f36" exitCode=0
Oct 30 21:25:38.325994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:38.325961    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6" event={"ID":"f6359677-6168-4783-9c33-494489293799","Type":"ContainerDied","Data":"69a96583497f6684f2dc16df0c3cd9d0e57414ba6443b6c1ef6e097b223f6f36"}
Oct 30 21:25:39.492439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.492407    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:39.628008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.627910    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca\") pod \"f6359677-6168-4783-9c33-494489293799\" (UID: \"f6359677-6168-4783-9c33-494489293799\") "
Oct 30 21:25:39.628008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.627971    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca\") pod \"f6359677-6168-4783-9c33-494489293799\" (UID: \"f6359677-6168-4783-9c33-494489293799\") "
Oct 30 21:25:39.628247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.628211    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca" pod "f6359677-6168-4783-9c33-494489293799" (UID: "f6359677-6168-4783-9c33-494489293799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:39.628247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.628239    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs\" (UniqueName: \"kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs\") pod \"f6359677-6168-4783-9c33-494489293799\" (UID: \"f6359677-6168-4783-9c33-494489293799\") "
Oct 30 21:25:39.628741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.628718    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6359677-6168-4783-9c33-494489293799-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:39.630096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.630069    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca" pod "f6359677-6168-4783-9c33-494489293799" (UID: "f6359677-6168-4783-9c33-494489293799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:39.630220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.630200    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs" pod "f6359677-6168-4783-9c33-494489293799" (UID: "f6359677-6168-4783-9c33-494489293799"). InnerVolumeSpecName "kube-api-access-5llbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:39.730475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.730435    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llbs\" (UniqueName: \"kubernetes.io/projected/f6359677-6168-4783-9c33-494489293799-kube-api-access-5llbs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:39.730475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:39.730469    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6359677-6168-4783-9c33-494489293799-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:40.376578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:40.376548    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"
Oct 30 21:25:40.376746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:40.376556    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6" event={"ID":"f6359677-6168-4783-9c33-494489293799","Type":"ContainerDied","Data":"ced733a7437f21bafe8efaa4179c1e2bd1e5e705aa673975db0e105b3981b975"}
Oct 30 21:25:40.376746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:40.376647    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced733a7437f21bafe8efaa4179c1e2bd1e5e705aa673975db0e105b3981b975"
Oct 30 21:25:42.198145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:42.198104    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:42.210662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:42.210632    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:42.401300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:42.401266    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:43.036624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.036568    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"]
Oct 30 21:25:43.037535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.037503    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f6359677-6168-4783-9c33-494489293799" containerName="revoke-keys"
Oct 30 21:25:43.037535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.037532    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6359677-6168-4783-9c33-494489293799" containerName="revoke-keys"
Oct 30 21:25:43.037815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.037731    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6359677-6168-4783-9c33-494489293799" containerName="revoke-keys"
Oct 30 21:25:43.042128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.042102    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.061895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.061863    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"]
Oct 30 21:25:43.183776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.183728    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.183937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.183802    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.183989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.183955    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.184022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.184006    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.184122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.184107    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.184223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.184208    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.184257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.184239    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddn8\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.184378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.184354    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.184431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.184409    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.285883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.285841    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.285902    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.285955    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286090    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286131    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddn8\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286197    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286233    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286262    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.286417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286303    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.287127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.286898    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.287290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.287263    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.287574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.287492    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.287574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.287553    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.287777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.287684    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.289126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.289065    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.289492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.289465    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.306084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.306057    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddn8\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.306282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.306258    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:43.359591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:43.359555    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:44.096920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:44.066861    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:44.096920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:44.066962    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:44.096920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:44.067003    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:44.213198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:44.212744    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"]
Oct 30 21:25:44.454113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:44.454023    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" event={"ID":"fae9c26d-4245-4118-a693-8315be8781b9","Type":"ContainerStarted","Data":"b05ab133b058420d3ccd08be010109df986d710308fcf82d65e30f9dddb6f889"}
Oct 30 21:25:45.026027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.025000    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:25:45.026027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.025719    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:25:45.053234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.053122    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:45.113562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.111722    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:45.113562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.112132    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:25:45.390807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.390704    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:25:45.492110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.492069    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" event={"ID":"fae9c26d-4245-4118-a693-8315be8781b9","Type":"ContainerStarted","Data":"3e6d1ca18800c6c9a6e02a37546b4b5bdb955159a6f4b6e22d4df80918400304"}
Oct 30 21:25:45.624628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:45.614694    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 30 21:25:46.061639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.057962    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.061639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.058709    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:46.111967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.111923    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.112571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.112548    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:25:46.124691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.113943    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.125257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.125231    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:25:46.128679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.127984    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podStartSLOduration=3.127967644 podStartE2EDuration="3.127967644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:45.605866528 +0000 UTC m=+928.528324831" watchObservedRunningTime="2026-10-30 21:25:46.127967644 +0000 UTC m=+929.050425943"
Oct 30 21:25:46.209447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.208242    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.236262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.233675    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:25:46.365471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.365336    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:46.404124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.403758    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:46.404124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.403840    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:46.407890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.407686    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.408206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:46.408073    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:47.100628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.095434    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.150629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.148330    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:25:47.171630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.170289    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:25:47.171824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.171672    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.264980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.264933    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:47.408882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.408768    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.446629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.446178    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:47.446629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.446237    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:47.963673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.963637    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:47.963999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:47.963950    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.108323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.108274    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.156054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.156009    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.157019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.156993    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:25:48.179217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.179172    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.212629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.194159    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.430802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.430761    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:48.431103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.431068    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:48.478630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.453197    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.479164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.457807    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.500791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.500739    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:48.501104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.501056    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.758629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.754699    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"]
Oct 30 21:25:48.853629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.850727    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:48.884632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.876627    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:48.884632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.876767    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:25:48.892630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.889417    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:48.892630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.889491    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.985736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.961731    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:48.985736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.961794    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.985736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.982743    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:48.985736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:48.982827    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.027238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.009982    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:49.027238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.010125    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:49.079846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.062992    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.079846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.063142    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2fh\" (UniqueName: \"kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.079846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.063213    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.080253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.080223    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"]
Oct 30 21:25:49.110935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.110871    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.182629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.157257    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.202631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.168507    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.202631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.189218    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2fh\" (UniqueName: \"kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.202631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.189297    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.202631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.190216    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.202631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.190810    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.251018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.250798    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.386879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.360029    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:49.386879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.360087    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:49.423008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.422971    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.499392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.499344    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:25:49.584481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.507429    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.742770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.734053    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:49.742770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.734136    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.779935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.779901    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:49.780224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.780194    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.781380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.781354    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2fh\" (UniqueName: \"kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh\") pod \"maas-api-revoke-sub-c30c14b3-rm7mf\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:49.788817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:49.788792    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:50.113874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.113800    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:25:50.207705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.200190    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.207705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.200624    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.267626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.266973    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.373494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.373043    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:50.373494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.373108    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:50.615241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.613770    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.621901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.621858    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.972713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:50.963619    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:25:51.132630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.130840    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:25:51.201719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.200876    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:51.201719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.201174    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:25:51.283647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.267220    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:25:51.493161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.493109    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:51.494064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.493707    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:51.707804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:51.707755    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:52.117629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.111485    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:52.288978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.288928    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:25:52.315529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.315458    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:25:52.316270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.316229    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:52.385685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.328119    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:52.385685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.328171    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:52.553628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.544248    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:52.553628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.544311    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:52.752622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.752562    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:52.775635 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:25:52.762945    6695 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.544s"
Oct 30 21:25:52.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.782821    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:52.839470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.782864    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:52.857210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:52.827157    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"]
Oct 30 21:25:53.000377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.000275    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf" event={"ID":"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1","Type":"ContainerStarted","Data":"be5569c7610486bbb021f763605573dcf02d4add379818f12596128cd63c50e8"}
Oct 30 21:25:53.295491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.295452    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:53.328590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.325716    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:53.380941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.352677    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:53.381565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.372066    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:53.381565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.381221    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:53.381565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.375720    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:53.381565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.373534    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:25:53.382594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.382577    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:25:53.410101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.410072    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:25:53.425631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.421220    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:25:53.431635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.429161    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:25:53.452631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.452574    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:25:53.467453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:53.457064    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:25:54.038730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:54.014797    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf" event={"ID":"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1","Type":"ContainerStarted","Data":"1e40aff0492317f3e8b8193650b763ca0eb16883690849d57ede48f4bc1a0487"}
Oct 30 21:25:54.055281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:54.055207    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf" podStartSLOduration=6.055187391 podStartE2EDuration="6.055187391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:54.054412562 +0000 UTC m=+936.976870863" watchObservedRunningTime="2026-10-30 21:25:54.055187391 +0000 UTC m=+936.977645690"
Oct 30 21:25:54.362680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:54.361699    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:54.362680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:54.361762    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:55.045359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:55.045327    6695 generic.go:356] "Generic (PLEG): container finished" podID="4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" containerID="1e40aff0492317f3e8b8193650b763ca0eb16883690849d57ede48f4bc1a0487" exitCode=0
Oct 30 21:25:55.045556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:55.045421    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf" event={"ID":"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1","Type":"ContainerDied","Data":"1e40aff0492317f3e8b8193650b763ca0eb16883690849d57ede48f4bc1a0487"}
Oct 30 21:25:55.365219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:55.365131    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:55.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:55.365782    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:56.239626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.239570    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:56.348082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.347999    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca\") pod \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") "
Oct 30 21:25:56.348082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.348064    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca\") pod \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") "
Oct 30 21:25:56.348256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.348208    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh\" (UniqueName: \"kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh\") pod \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\" (UID: \"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1\") "
Oct 30 21:25:56.348427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.348406    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca" pod "4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" (UID: "4bebea0c-a750-4a7e-87b6-0bab7f90ecb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:56.349045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.349000    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:56.350286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.350263    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca" pod "4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" (UID: "4bebea0c-a750-4a7e-87b6-0bab7f90ecb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:56.350378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.350323    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh" pod "4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" (UID: "4bebea0c-a750-4a7e-87b6-0bab7f90ecb1"). InnerVolumeSpecName "kube-api-access-2g2fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:56.360719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.360697    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:25:56.360817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.360746    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 21:25:56.450148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.450122    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:56.450148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.450148    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2fh\" (UniqueName: \"kubernetes.io/projected/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1-kube-api-access-2g2fh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:25:56.824511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.824475    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"]
Oct 30 21:25:56.825105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.825089    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" containerName="revoke-keys"
Oct 30 21:25:56.825193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.825105    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" containerName="revoke-keys"
Oct 30 21:25:56.825252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.825198    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" containerName="revoke-keys"
Oct 30 21:25:56.828812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.828793    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:56.849644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.849596    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"]
Oct 30 21:25:56.955794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.955752    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxw8j\" (UniqueName: \"kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:56.955794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.955792    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:56.956042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:56.955874    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.061627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.057914    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxw8j\" (UniqueName: \"kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.061627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.058349    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.061627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.058417    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.061627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.059461    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.080681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.064728    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.083868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.082240    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf" event={"ID":"4bebea0c-a750-4a7e-87b6-0bab7f90ecb1","Type":"ContainerDied","Data":"be5569c7610486bbb021f763605573dcf02d4add379818f12596128cd63c50e8"}
Oct 30 21:25:57.083868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.082311    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be5569c7610486bbb021f763605573dcf02d4add379818f12596128cd63c50e8"
Oct 30 21:25:57.083868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.082380    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"
Oct 30 21:25:57.087330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.084634    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxw8j\" (UniqueName: \"kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j\") pod \"maas-api-revoke-sub-e6d62e33-bjd5n\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.145282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.145240    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:25:57.474959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.473723    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"]
Oct 30 21:25:57.477088 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:57.477057    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eff7c38_5dc0_4f75_b722_6f36be69687c.slice/crio-30aa8993b943ea2e4c2d9c0d87e8eabac814c7d1bee0117713e0e90c88844ade WatchSource:0}: Error finding container 30aa8993b943ea2e4c2d9c0d87e8eabac814c7d1bee0117713e0e90c88844ade: Status 404 returned error can't find the container with id 30aa8993b943ea2e4c2d9c0d87e8eabac814c7d1bee0117713e0e90c88844ade
Oct 30 21:25:57.522046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.522015    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"]
Oct 30 21:25:57.527647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.527625    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.544788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.544762    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"]
Oct 30 21:25:57.630275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.630219    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:57.630884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.630857    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:57.632140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.632118    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:25:57.673594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.673508    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.673793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.673721    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8hf\" (UniqueName: \"kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.673927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.673905    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.775323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.775285    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.775533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.775503    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.775698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.775589    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8hf\" (UniqueName: \"kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.776051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.776023    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.777947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.777925    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.793993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.793965    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8hf\" (UniqueName: \"kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf\") pod \"maas-api-revoke-sub-6ce1c589-cxxvr\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:57.852866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:57.852835    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:25:58.099283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:58.099234    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n" event={"ID":"1eff7c38-5dc0-4f75-b722-6f36be69687c","Type":"ContainerStarted","Data":"a6c3a553db78ec27ab0d362a0b09560aacebb0e9fa7ead582a93106bfe6e01b6"}
Oct 30 21:25:58.099449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:58.099354    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n" event={"ID":"1eff7c38-5dc0-4f75-b722-6f36be69687c","Type":"ContainerStarted","Data":"30aa8993b943ea2e4c2d9c0d87e8eabac814c7d1bee0117713e0e90c88844ade"}
Oct 30 21:25:58.139034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:58.138981    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n" podStartSLOduration=2.138967925 podStartE2EDuration="2.138967925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:58.135805109 +0000 UTC m=+941.058263406" watchObservedRunningTime="2026-10-30 21:25:58.138967925 +0000 UTC m=+941.061426258"
Oct 30 21:25:58.152464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:58.152437    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"]
Oct 30 21:25:58.155857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:25:58.155832    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8781ebe8_db2b_4577_b22c_6a16447a6586.slice/crio-3ff6136a93c17271ae9093d2bb52eb8a862391e2271c42281a0626340c42288d WatchSource:0}: Error finding container 3ff6136a93c17271ae9093d2bb52eb8a862391e2271c42281a0626340c42288d: Status 404 returned error can't find the container with id 3ff6136a93c17271ae9093d2bb52eb8a862391e2271c42281a0626340c42288d
Oct 30 21:25:59.107532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:59.107493    6695 generic.go:356] "Generic (PLEG): container finished" podID="8781ebe8-db2b-4577-b22c-6a16447a6586" containerID="d067ac286572a2b6f22f329d5149596d04209afdba44a8d1e2f104ddd11e1df1" exitCode=0
Oct 30 21:25:59.108018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:59.107579    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr" event={"ID":"8781ebe8-db2b-4577-b22c-6a16447a6586","Type":"ContainerDied","Data":"d067ac286572a2b6f22f329d5149596d04209afdba44a8d1e2f104ddd11e1df1"}
Oct 30 21:25:59.108018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:59.107646    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr" event={"ID":"8781ebe8-db2b-4577-b22c-6a16447a6586","Type":"ContainerStarted","Data":"3ff6136a93c17271ae9093d2bb52eb8a862391e2271c42281a0626340c42288d"}
Oct 30 21:25:59.109346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:59.109319    6695 generic.go:356] "Generic (PLEG): container finished" podID="1eff7c38-5dc0-4f75-b722-6f36be69687c" containerID="a6c3a553db78ec27ab0d362a0b09560aacebb0e9fa7ead582a93106bfe6e01b6" exitCode=0
Oct 30 21:25:59.109473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:25:59.109359    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n" event={"ID":"1eff7c38-5dc0-4f75-b722-6f36be69687c","Type":"ContainerDied","Data":"a6c3a553db78ec27ab0d362a0b09560aacebb0e9fa7ead582a93106bfe6e01b6"}
Oct 30 21:26:00.327344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.327310    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:26:00.329767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.329746    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:26:00.423708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.420084    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca\") pod \"8781ebe8-db2b-4577-b22c-6a16447a6586\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") "
Oct 30 21:26:00.423708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.420176    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf\" (UniqueName: \"kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf\") pod \"8781ebe8-db2b-4577-b22c-6a16447a6586\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") "
Oct 30 21:26:00.423708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.420259    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j\" (UniqueName: \"kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j\") pod \"1eff7c38-5dc0-4f75-b722-6f36be69687c\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") "
Oct 30 21:26:00.423708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.420304    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca\") pod \"8781ebe8-db2b-4577-b22c-6a16447a6586\" (UID: \"8781ebe8-db2b-4577-b22c-6a16447a6586\") "
Oct 30 21:26:00.423708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.420444    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca\") pod \"1eff7c38-5dc0-4f75-b722-6f36be69687c\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") "
Oct 30 21:26:00.423708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.420510    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca\") pod \"1eff7c38-5dc0-4f75-b722-6f36be69687c\" (UID: \"1eff7c38-5dc0-4f75-b722-6f36be69687c\") "
Oct 30 21:26:00.457198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.425318    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca" pod "1eff7c38-5dc0-4f75-b722-6f36be69687c" (UID: "1eff7c38-5dc0-4f75-b722-6f36be69687c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:00.457198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.453690    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca" pod "8781ebe8-db2b-4577-b22c-6a16447a6586" (UID: "8781ebe8-db2b-4577-b22c-6a16447a6586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:00.457198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.456366    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf" pod "8781ebe8-db2b-4577-b22c-6a16447a6586" (UID: "8781ebe8-db2b-4577-b22c-6a16447a6586"). InnerVolumeSpecName "kube-api-access-pf8hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:00.457198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.456961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca" pod "8781ebe8-db2b-4577-b22c-6a16447a6586" (UID: "8781ebe8-db2b-4577-b22c-6a16447a6586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:00.462701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.460194    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j" pod "1eff7c38-5dc0-4f75-b722-6f36be69687c" (UID: "1eff7c38-5dc0-4f75-b722-6f36be69687c"). InnerVolumeSpecName "kube-api-access-dxw8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.463686    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"]
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.464564    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1eff7c38-5dc0-4f75-b722-6f36be69687c" containerName="revoke-keys"
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.464581    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eff7c38-5dc0-4f75-b722-6f36be69687c" containerName="revoke-keys"
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.464639    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8781ebe8-db2b-4577-b22c-6a16447a6586" containerName="revoke-keys"
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.464648    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8781ebe8-db2b-4577-b22c-6a16447a6586" containerName="revoke-keys"
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.464795    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8781ebe8-db2b-4577-b22c-6a16447a6586" containerName="revoke-keys"
Oct 30 21:26:00.464933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.464812    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eff7c38-5dc0-4f75-b722-6f36be69687c" containerName="revoke-keys"
Oct 30 21:26:00.465775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.465739    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca" pod "1eff7c38-5dc0-4f75-b722-6f36be69687c" (UID: "1eff7c38-5dc0-4f75-b722-6f36be69687c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:00.473973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.473946    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.495533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.495493    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"]
Oct 30 21:26:00.522711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.522670    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcccj\" (UniqueName: \"kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.522946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.522927    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.523051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523036    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.523176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523162    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.523225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523204    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.523279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523237    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf8hf\" (UniqueName: \"kubernetes.io/projected/8781ebe8-db2b-4577-b22c-6a16447a6586-kube-api-access-pf8hf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.523279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523255    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxw8j\" (UniqueName: \"kubernetes.io/projected/1eff7c38-5dc0-4f75-b722-6f36be69687c-kube-api-access-dxw8j\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.523279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523270    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8781ebe8-db2b-4577-b22c-6a16447a6586-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.523372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.523286    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eff7c38-5dc0-4f75-b722-6f36be69687c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:00.624557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.624517    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcccj\" (UniqueName: \"kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.624788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.624679    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.624788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.624776    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.625387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.625362    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.627141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.627122    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.642799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.642773    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcccj\" (UniqueName: \"kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj\") pod \"maas-api-revoke-sub-a33decec-rlwwc\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:00.810094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:00.810055    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:01.075567 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:01.075475    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:26:01.117331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.113229    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"]
Oct 30 21:26:01.118656 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:01.118594    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1562005_296e_4cb8_8870_7e7dec936f6a.slice/crio-78dc70ce9009d55dd71c47ddf10d16698686c85d6f4a01a8656b609cf3c26393 WatchSource:0}: Error finding container 78dc70ce9009d55dd71c47ddf10d16698686c85d6f4a01a8656b609cf3c26393: Status 404 returned error can't find the container with id 78dc70ce9009d55dd71c47ddf10d16698686c85d6f4a01a8656b609cf3c26393
Oct 30 21:26:01.131495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.131473    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"
Oct 30 21:26:01.131628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.131480    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n" event={"ID":"1eff7c38-5dc0-4f75-b722-6f36be69687c","Type":"ContainerDied","Data":"30aa8993b943ea2e4c2d9c0d87e8eabac814c7d1bee0117713e0e90c88844ade"}
Oct 30 21:26:01.131693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.131595    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30aa8993b943ea2e4c2d9c0d87e8eabac814c7d1bee0117713e0e90c88844ade"
Oct 30 21:26:01.133046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.133021    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc" event={"ID":"c1562005-296e-4cb8-8870-7e7dec936f6a","Type":"ContainerStarted","Data":"78dc70ce9009d55dd71c47ddf10d16698686c85d6f4a01a8656b609cf3c26393"}
Oct 30 21:26:01.134574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.134550    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr" event={"ID":"8781ebe8-db2b-4577-b22c-6a16447a6586","Type":"ContainerDied","Data":"3ff6136a93c17271ae9093d2bb52eb8a862391e2271c42281a0626340c42288d"}
Oct 30 21:26:01.134701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.134582    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff6136a93c17271ae9093d2bb52eb8a862391e2271c42281a0626340c42288d"
Oct 30 21:26:01.134701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:01.134584    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"
Oct 30 21:26:02.169626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:02.161137    6695 generic.go:356] "Generic (PLEG): container finished" podID="c1562005-296e-4cb8-8870-7e7dec936f6a" containerID="809d1c5385d26fec5d10c21fb5f79aa0554b549b382ddcfad9c3f5713ffd99f3" exitCode=0
Oct 30 21:26:02.169626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:02.161387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc" event={"ID":"c1562005-296e-4cb8-8870-7e7dec936f6a","Type":"ContainerDied","Data":"809d1c5385d26fec5d10c21fb5f79aa0554b549b382ddcfad9c3f5713ffd99f3"}
Oct 30 21:26:04.047629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.047269    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:04.097457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.096311    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca\") pod \"c1562005-296e-4cb8-8870-7e7dec936f6a\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") "
Oct 30 21:26:04.097457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.096369    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca\") pod \"c1562005-296e-4cb8-8870-7e7dec936f6a\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") "
Oct 30 21:26:04.097457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.096486    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj\" (UniqueName: \"kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj\") pod \"c1562005-296e-4cb8-8870-7e7dec936f6a\" (UID: \"c1562005-296e-4cb8-8870-7e7dec936f6a\") "
Oct 30 21:26:04.126628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.116692    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca" pod "c1562005-296e-4cb8-8870-7e7dec936f6a" (UID: "c1562005-296e-4cb8-8870-7e7dec936f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:04.126959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.126907    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca" pod "c1562005-296e-4cb8-8870-7e7dec936f6a" (UID: "c1562005-296e-4cb8-8870-7e7dec936f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:04.158334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.157179    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj" pod "c1562005-296e-4cb8-8870-7e7dec936f6a" (UID: "c1562005-296e-4cb8-8870-7e7dec936f6a"). InnerVolumeSpecName "kube-api-access-dcccj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:04.258723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.252421    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcccj\" (UniqueName: \"kubernetes.io/projected/c1562005-296e-4cb8-8870-7e7dec936f6a-kube-api-access-dcccj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:04.258723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.252460    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:04.258723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.252478    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1562005-296e-4cb8-8870-7e7dec936f6a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:04.281658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.269645    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc" event={"ID":"c1562005-296e-4cb8-8870-7e7dec936f6a","Type":"ContainerDied","Data":"78dc70ce9009d55dd71c47ddf10d16698686c85d6f4a01a8656b609cf3c26393"}
Oct 30 21:26:04.281658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.269713    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78dc70ce9009d55dd71c47ddf10d16698686c85d6f4a01a8656b609cf3c26393"
Oct 30 21:26:04.284646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:04.284621    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"
Oct 30 21:26:05.128741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.128694    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 30 21:26:05.188811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.188702    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:05.189422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.189386    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:05.236835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.236783    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:05.467335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.465090    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:05.634620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.598895    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:26:05.640981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.635691    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:05.672324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.672291    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"]
Oct 30 21:26:05.679528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.679494    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c1562005-296e-4cb8-8870-7e7dec936f6a" containerName="revoke-keys"
Oct 30 21:26:05.679755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.679736    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1562005-296e-4cb8-8870-7e7dec936f6a" containerName="revoke-keys"
Oct 30 21:26:05.680013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.679998    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1562005-296e-4cb8-8870-7e7dec936f6a" containerName="revoke-keys"
Oct 30 21:26:05.689000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.688967    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"]
Oct 30 21:26:05.689312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.689291    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.739982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.739300    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:05.740278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.739770    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:05.740392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.739824    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:26:05.740468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.740232    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:05.784500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.784447    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.784919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.784895    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.785116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.785099    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvk8\" (UniqueName: \"kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.887465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.887422    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.887685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.887496    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.887685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.887573    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvk8\" (UniqueName: \"kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.892624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.888899    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.924230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.902384    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:05.924230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:05.921262    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvk8\" (UniqueName: \"kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8\") pod \"maas-api-revoke-sub-43248925-xrz9p\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:06.050249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:06.050212    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:06.356012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:06.355988    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"]
Oct 30 21:26:06.358829 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:06.358797    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688b26a3_59ab_412f_ab65_ed2db67d7083.slice/crio-e6b76915ac03d777a2413c6fe5d219cc297e7d4db416a64715d05f9a044d4243 WatchSource:0}: Error finding container e6b76915ac03d777a2413c6fe5d219cc297e7d4db416a64715d05f9a044d4243: Status 404 returned error can't find the container with id e6b76915ac03d777a2413c6fe5d219cc297e7d4db416a64715d05f9a044d4243
Oct 30 21:26:06.748725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:06.748678    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p" event={"ID":"688b26a3-59ab-412f-ab65-ed2db67d7083","Type":"ContainerStarted","Data":"e301c5d6ffc07ff52cc3624b6ee60dcc5f858ff88ea5ef8725e059b3747df675"}
Oct 30 21:26:06.748922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:06.748777    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p" event={"ID":"688b26a3-59ab-412f-ab65-ed2db67d7083","Type":"ContainerStarted","Data":"e6b76915ac03d777a2413c6fe5d219cc297e7d4db416a64715d05f9a044d4243"}
Oct 30 21:26:06.789474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:06.789422    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p" podStartSLOduration=1.789406333 podStartE2EDuration="1.789406333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:06.785440127 +0000 UTC m=+949.707898424" watchObservedRunningTime="2026-10-30 21:26:06.789406333 +0000 UTC m=+949.711864632"
Oct 30 21:26:07.757184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:07.757147    6695 generic.go:356] "Generic (PLEG): container finished" podID="688b26a3-59ab-412f-ab65-ed2db67d7083" containerID="e301c5d6ffc07ff52cc3624b6ee60dcc5f858ff88ea5ef8725e059b3747df675" exitCode=0
Oct 30 21:26:07.757580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:07.757229    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p" event={"ID":"688b26a3-59ab-412f-ab65-ed2db67d7083","Type":"ContainerDied","Data":"e301c5d6ffc07ff52cc3624b6ee60dcc5f858ff88ea5ef8725e059b3747df675"}
Oct 30 21:26:07.856939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:07.856902    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"]
Oct 30 21:26:07.861505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:07.861484    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:07.877168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:07.877128    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"]
Oct 30 21:26:08.016466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.016387    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.016635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.016585    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxnd\" (UniqueName: \"kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.016756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.016737    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.118005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.117974    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxnd\" (UniqueName: \"kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.118183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.118040    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.118183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.118109    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.118819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.118787    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.120437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.120410    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.136271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.136246    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxnd\" (UniqueName: \"kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd\") pod \"maas-api-revoke-sub-77b7de24-7mn7x\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.176644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.176599    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:08.479048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.479025    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"]
Oct 30 21:26:08.481531 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:08.481505    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94cba4_80b7_4a90_84f5_c9be9494b031.slice/crio-22d47f832460fa4b88e238e358291799bbb561b1843ec4a298cd4a7553534a15 WatchSource:0}: Error finding container 22d47f832460fa4b88e238e358291799bbb561b1843ec4a298cd4a7553534a15: Status 404 returned error can't find the container with id 22d47f832460fa4b88e238e358291799bbb561b1843ec4a298cd4a7553534a15
Oct 30 21:26:08.768027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.767989    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x" event={"ID":"dc94cba4-80b7-4a90-84f5-c9be9494b031","Type":"ContainerStarted","Data":"22d47f832460fa4b88e238e358291799bbb561b1843ec4a298cd4a7553534a15"}
Oct 30 21:26:08.923420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:08.918174    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:09.046243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.046206    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca\") pod \"688b26a3-59ab-412f-ab65-ed2db67d7083\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") "
Oct 30 21:26:09.046432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.046315    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca\") pod \"688b26a3-59ab-412f-ab65-ed2db67d7083\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") "
Oct 30 21:26:09.046432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.046423    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8\" (UniqueName: \"kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8\") pod \"688b26a3-59ab-412f-ab65-ed2db67d7083\" (UID: \"688b26a3-59ab-412f-ab65-ed2db67d7083\") "
Oct 30 21:26:09.046836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.046807    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca" pod "688b26a3-59ab-412f-ab65-ed2db67d7083" (UID: "688b26a3-59ab-412f-ab65-ed2db67d7083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:09.046977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.046941    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:09.048278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.048254    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca" pod "688b26a3-59ab-412f-ab65-ed2db67d7083" (UID: "688b26a3-59ab-412f-ab65-ed2db67d7083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:09.048411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.048390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8" pod "688b26a3-59ab-412f-ab65-ed2db67d7083" (UID: "688b26a3-59ab-412f-ab65-ed2db67d7083"). InnerVolumeSpecName "kube-api-access-ltvk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:09.148206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.148176    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvk8\" (UniqueName: \"kubernetes.io/projected/688b26a3-59ab-412f-ab65-ed2db67d7083-kube-api-access-ltvk8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:09.148206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.148204    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/688b26a3-59ab-412f-ab65-ed2db67d7083-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:09.780026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.779996    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"
Oct 30 21:26:09.783901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.783870    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p" event={"ID":"688b26a3-59ab-412f-ab65-ed2db67d7083","Type":"ContainerDied","Data":"e6b76915ac03d777a2413c6fe5d219cc297e7d4db416a64715d05f9a044d4243"}
Oct 30 21:26:09.784319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.784289    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b76915ac03d777a2413c6fe5d219cc297e7d4db416a64715d05f9a044d4243"
Oct 30 21:26:09.786031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.786011    6695 generic.go:356] "Generic (PLEG): container finished" podID="dc94cba4-80b7-4a90-84f5-c9be9494b031" containerID="12c1269bc4ce4393d547567773c42f06b432fdbd225645c9b52a07b5fed2f507" exitCode=0
Oct 30 21:26:09.786243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:09.786223    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x" event={"ID":"dc94cba4-80b7-4a90-84f5-c9be9494b031","Type":"ContainerDied","Data":"12c1269bc4ce4393d547567773c42f06b432fdbd225645c9b52a07b5fed2f507"}
Oct 30 21:26:10.907427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.907387    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"]
Oct 30 21:26:10.908223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.908200    6695 cpu_manager.go:414] "containerMap: removing container" podUID="688b26a3-59ab-412f-ab65-ed2db67d7083" containerName="revoke-keys"
Oct 30 21:26:10.908223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.908225    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688b26a3-59ab-412f-ab65-ed2db67d7083" containerName="revoke-keys"
Oct 30 21:26:10.908404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.908392    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="688b26a3-59ab-412f-ab65-ed2db67d7083" containerName="revoke-keys"
Oct 30 21:26:10.912518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.912495    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:10.929255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.929227    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"]
Oct 30 21:26:10.948959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.948930    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:10.986822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.986785    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd\" (UniqueName: \"kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd\") pod \"dc94cba4-80b7-4a90-84f5-c9be9494b031\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") "
Oct 30 21:26:10.986990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.986977    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca\") pod \"dc94cba4-80b7-4a90-84f5-c9be9494b031\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") "
Oct 30 21:26:10.987048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.987016    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca\") pod \"dc94cba4-80b7-4a90-84f5-c9be9494b031\" (UID: \"dc94cba4-80b7-4a90-84f5-c9be9494b031\") "
Oct 30 21:26:10.987437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.987411    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:10.987760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.987738    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:10.988240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.988057    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96n5q\" (UniqueName: \"kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:10.988409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.988067    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca" pod "dc94cba4-80b7-4a90-84f5-c9be9494b031" (UID: "dc94cba4-80b7-4a90-84f5-c9be9494b031"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:10.990283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.990156    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd" pod "dc94cba4-80b7-4a90-84f5-c9be9494b031" (UID: "dc94cba4-80b7-4a90-84f5-c9be9494b031"). InnerVolumeSpecName "kube-api-access-hnxnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:10.990899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:10.990873    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca" pod "dc94cba4-80b7-4a90-84f5-c9be9494b031" (UID: "dc94cba4-80b7-4a90-84f5-c9be9494b031"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:11.089365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.089276    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.089543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.089387    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.089543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.089456    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96n5q\" (UniqueName: \"kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.089697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.089642    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:11.089697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.089663    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc94cba4-80b7-4a90-84f5-c9be9494b031-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:11.089697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.089678    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnxnd\" (UniqueName: \"kubernetes.io/projected/dc94cba4-80b7-4a90-84f5-c9be9494b031-kube-api-access-hnxnd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:11.090186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.090158    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.091727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.091707    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.108172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.108143    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96n5q\" (UniqueName: \"kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q\") pod \"maas-api-revoke-sub-69b4ea00-kxnsc\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.230234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.230197    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:11.531101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.531033    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"]
Oct 30 21:26:11.533629 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:11.533586    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98c7fc3_b122_4bff_bb9d_1c8471f0223e.slice/crio-afbca01c19369ab7cb384d0616508dcfdf93df2a34f635753b9fd44a15799e37 WatchSource:0}: Error finding container afbca01c19369ab7cb384d0616508dcfdf93df2a34f635753b9fd44a15799e37: Status 404 returned error can't find the container with id afbca01c19369ab7cb384d0616508dcfdf93df2a34f635753b9fd44a15799e37
Oct 30 21:26:11.801494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.801405    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"
Oct 30 21:26:11.801494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.801418    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x" event={"ID":"dc94cba4-80b7-4a90-84f5-c9be9494b031","Type":"ContainerDied","Data":"22d47f832460fa4b88e238e358291799bbb561b1843ec4a298cd4a7553534a15"}
Oct 30 21:26:11.801494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.801469    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22d47f832460fa4b88e238e358291799bbb561b1843ec4a298cd4a7553534a15"
Oct 30 21:26:11.803082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:11.803058    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc" event={"ID":"a98c7fc3-b122-4bff-bb9d-1c8471f0223e","Type":"ContainerStarted","Data":"afbca01c19369ab7cb384d0616508dcfdf93df2a34f635753b9fd44a15799e37"}
Oct 30 21:26:12.476115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.474318    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"]
Oct 30 21:26:12.485255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.485222    6695 cpu_manager.go:414] "containerMap: removing container" podUID="dc94cba4-80b7-4a90-84f5-c9be9494b031" containerName="revoke-keys"
Oct 30 21:26:12.485449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.485433    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc94cba4-80b7-4a90-84f5-c9be9494b031" containerName="revoke-keys"
Oct 30 21:26:12.485780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.485766    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc94cba4-80b7-4a90-84f5-c9be9494b031" containerName="revoke-keys"
Oct 30 21:26:12.490053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.490031    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"]
Oct 30 21:26:12.490209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.490126    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.505081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.505054    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:26:12.505209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.505113    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:26:12.505209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.505125    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:26:12.505209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.505163    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d93cf183177\""
Oct 30 21:26:12.505209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.505167    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:26:12.516394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.516374    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dmwj5\""
Oct 30 21:26:12.645911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.645876    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.646112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.646090    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.646170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.646138    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.646241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.646229    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrf9p\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.747665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.747542    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrf9p\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.747825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.747715    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.747918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.747855    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.747918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.747900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.748527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.748502    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.748652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.748590    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.750163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.750144    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.766615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.766586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrf9p\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p\") pod \"payload-processing-external-model-d4877f77-k9hwm\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") " pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:12.875628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.871445    6695 generic.go:356] "Generic (PLEG): container finished" podID="a98c7fc3-b122-4bff-bb9d-1c8471f0223e" containerID="028cbfcb72c60e3fbacc394c4a9a6f980ac81b73a4899163fd44ae987cd47843" exitCode=0
Oct 30 21:26:12.875628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.871574    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc" event={"ID":"a98c7fc3-b122-4bff-bb9d-1c8471f0223e","Type":"ContainerDied","Data":"028cbfcb72c60e3fbacc394c4a9a6f980ac81b73a4899163fd44ae987cd47843"}
Oct 30 21:26:12.875628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:12.872336    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:13.127899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.127867    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"]
Oct 30 21:26:13.133394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.133369    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.154885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.154853    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"]
Oct 30 21:26:13.177196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.177164    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"]
Oct 30 21:26:13.182421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.182394    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.244773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.244733    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"]
Oct 30 21:26:13.257619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.257573    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwt9\" (UniqueName: \"kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.257764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.257709    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.257809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.257772    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44m2\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.257851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.257837    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.258052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258024    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.258162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258086    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.258162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258154    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.258296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258274    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.258626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258577    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.258720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258659    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.258720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258685    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.258808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.258753    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.301023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.301000    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"]
Oct 30 21:26:13.303780 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:13.303729    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ef6d10_d973_4031_9f74_80a6d305f576.slice/crio-f5c7b5945e90a1099161013e8601b7f598d00597564284dffdf710d949f00e18 WatchSource:0}: Error finding container f5c7b5945e90a1099161013e8601b7f598d00597564284dffdf710d949f00e18: Status 404 returned error can't find the container with id f5c7b5945e90a1099161013e8601b7f598d00597564284dffdf710d949f00e18
Oct 30 21:26:13.360062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360030    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360076    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360114    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360156    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360208    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360240    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.360311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360267    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.360311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360307    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360351    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwt9\" (UniqueName: \"kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.360492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360402    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44m2\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.360668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360652    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.361087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360652    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.361087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.360977    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.361219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.361100    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.361937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.361649    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.361937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.361900    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.363542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.363301    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.364213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.364195    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.364967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.364943    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.378455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.378429    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.378910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.378888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44m2\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.380180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.380159    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwt9\" (UniqueName: \"kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9\") pod \"maas-api-revoke-sub-c638681d-cbq6k\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.465594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.465546    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:13.521114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.521078    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:13.728179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.728137    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"]
Oct 30 21:26:13.733596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.733572    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.751026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.751002    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"]
Oct 30 21:26:13.796481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.796458    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"]
Oct 30 21:26:13.799038 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:13.799007    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b69f28c_173a_497d_84c5_d07cce3a16fb.slice/crio-3a24fad80078198a39b9f72e1dd70fc19d946ee9228042286dcd23f9661b5f2e WatchSource:0}: Error finding container 3a24fad80078198a39b9f72e1dd70fc19d946ee9228042286dcd23f9661b5f2e: Status 404 returned error can't find the container with id 3a24fad80078198a39b9f72e1dd70fc19d946ee9228042286dcd23f9661b5f2e
Oct 30 21:26:13.801661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.801629    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:13.801717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.801697    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:13.801748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.801726    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:13.843082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.843054    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"]
Oct 30 21:26:13.844657 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:13.844628    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4b0601_8bf6_41d8_a382_a70121071475.slice/crio-b5a0a61ac052c70703c8d29ae7124559cfbda5dfc9c9e4c1ace3ae97b6d2a621 WatchSource:0}: Error finding container b5a0a61ac052c70703c8d29ae7124559cfbda5dfc9c9e4c1ace3ae97b6d2a621: Status 404 returned error can't find the container with id b5a0a61ac052c70703c8d29ae7124559cfbda5dfc9c9e4c1ace3ae97b6d2a621
Oct 30 21:26:13.872534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.872509    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.872648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.872544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9pq\" (UniqueName: \"kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.872810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.872786    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.881012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.880980    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k" event={"ID":"4f4b0601-8bf6-41d8-a382-a70121071475","Type":"ContainerStarted","Data":"b5a0a61ac052c70703c8d29ae7124559cfbda5dfc9c9e4c1ace3ae97b6d2a621"}
Oct 30 21:26:13.882810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.882781    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm" event={"ID":"12ef6d10-d973-4031-9f74-80a6d305f576","Type":"ContainerStarted","Data":"af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137"}
Oct 30 21:26:13.882933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.882835    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm" event={"ID":"12ef6d10-d973-4031-9f74-80a6d305f576","Type":"ContainerStarted","Data":"f5c7b5945e90a1099161013e8601b7f598d00597564284dffdf710d949f00e18"}
Oct 30 21:26:13.882933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.882868    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:13.884473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.884451    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" event={"ID":"7b69f28c-173a-497d-84c5-d07cce3a16fb","Type":"ContainerStarted","Data":"091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45"}
Oct 30 21:26:13.884559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.884489    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" event={"ID":"7b69f28c-173a-497d-84c5-d07cce3a16fb","Type":"ContainerStarted","Data":"3a24fad80078198a39b9f72e1dd70fc19d946ee9228042286dcd23f9661b5f2e"}
Oct 30 21:26:13.924051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.923987    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm" podStartSLOduration=1.9239670530000001 podStartE2EDuration="1.923967053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:13.920969593 +0000 UTC m=+956.843427892" watchObservedRunningTime="2026-10-30 21:26:13.923967053 +0000 UTC m=+956.846425357"
Oct 30 21:26:13.965641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.965553    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podStartSLOduration=0.965536824 podStartE2EDuration="965.536824ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:13.961341799 +0000 UTC m=+956.883800108" watchObservedRunningTime="2026-10-30 21:26:13.965536824 +0000 UTC m=+956.887995121"
Oct 30 21:26:13.974200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.974163    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.974637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.974590    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.974724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.974652    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9pq\" (UniqueName: \"kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:13.975803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:13.975721    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:14.108700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.105150    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:14.114926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.113027    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9pq\" (UniqueName: \"kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq\") pod \"maas-api-revoke-sub-b30e39f1-xg8wf\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:14.176988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.176959    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:14.288163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.288125    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca\") pod \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") "
Oct 30 21:26:14.288348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.288201    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q\" (UniqueName: \"kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q\") pod \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") "
Oct 30 21:26:14.288414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.288346    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca\") pod \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\" (UID: \"a98c7fc3-b122-4bff-bb9d-1c8471f0223e\") "
Oct 30 21:26:14.288488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.288459    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca" pod "a98c7fc3-b122-4bff-bb9d-1c8471f0223e" (UID: "a98c7fc3-b122-4bff-bb9d-1c8471f0223e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:14.288842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.288810    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:14.290247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.290226    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q" pod "a98c7fc3-b122-4bff-bb9d-1c8471f0223e" (UID: "a98c7fc3-b122-4bff-bb9d-1c8471f0223e"). InnerVolumeSpecName "kube-api-access-96n5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:14.290352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.290333    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca" pod "a98c7fc3-b122-4bff-bb9d-1c8471f0223e" (UID: "a98c7fc3-b122-4bff-bb9d-1c8471f0223e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:14.351086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.351057    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:14.393663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.389905    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96n5q\" (UniqueName: \"kubernetes.io/projected/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-kube-api-access-96n5q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:14.393663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.389938    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a98c7fc3-b122-4bff-bb9d-1c8471f0223e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:14.479261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.479005    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:14.481947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.481913    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:14.482064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.481970    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:14.677377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.677343    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"]
Oct 30 21:26:14.679553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:14.679526    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad1d93c_7486_4e99_8a28_108dbf4b659d.slice/crio-48b608612a4e2b436454a469dcb344ca1f86fb04bacc1aac15fee94c203579c9 WatchSource:0}: Error finding container 48b608612a4e2b436454a469dcb344ca1f86fb04bacc1aac15fee94c203579c9: Status 404 returned error can't find the container with id 48b608612a4e2b436454a469dcb344ca1f86fb04bacc1aac15fee94c203579c9
Oct 30 21:26:14.892874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.892838    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf" event={"ID":"dad1d93c-7486-4e99-8a28-108dbf4b659d","Type":"ContainerStarted","Data":"48b608612a4e2b436454a469dcb344ca1f86fb04bacc1aac15fee94c203579c9"}
Oct 30 21:26:14.894543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.894516    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k" event={"ID":"4f4b0601-8bf6-41d8-a382-a70121071475","Type":"ContainerStarted","Data":"82d2e9d444145c42f269cfbebda1e5c18fc62d447716021136fab5fd67683fbe"}
Oct 30 21:26:14.896051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.896032    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"
Oct 30 21:26:14.896144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.896071    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc" event={"ID":"a98c7fc3-b122-4bff-bb9d-1c8471f0223e","Type":"ContainerDied","Data":"afbca01c19369ab7cb384d0616508dcfdf93df2a34f635753b9fd44a15799e37"}
Oct 30 21:26:14.896144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.896111    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbca01c19369ab7cb384d0616508dcfdf93df2a34f635753b9fd44a15799e37"
Oct 30 21:26:14.942384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:14.942282    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k" podStartSLOduration=1.942268255 podStartE2EDuration="1.942268255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:14.937879767 +0000 UTC m=+957.860338058" watchObservedRunningTime="2026-10-30 21:26:14.942268255 +0000 UTC m=+957.864726583"
Oct 30 21:26:15.476633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:15.473015    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:15.476633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:15.473077    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:15.964705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:15.959808    6695 generic.go:356] "Generic (PLEG): container finished" podID="4f4b0601-8bf6-41d8-a382-a70121071475" containerID="82d2e9d444145c42f269cfbebda1e5c18fc62d447716021136fab5fd67683fbe" exitCode=0
Oct 30 21:26:15.964705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:15.960025    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k" event={"ID":"4f4b0601-8bf6-41d8-a382-a70121071475","Type":"ContainerDied","Data":"82d2e9d444145c42f269cfbebda1e5c18fc62d447716021136fab5fd67683fbe"}
Oct 30 21:26:16.000013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:15.999969    6695 generic.go:356] "Generic (PLEG): container finished" podID="dad1d93c-7486-4e99-8a28-108dbf4b659d" containerID="aa6fbfd346067de0ebac86051f90cbd1cf42572bc21e13c0cdf7225702e99dd6" exitCode=0
Oct 30 21:26:16.000467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:16.000437    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf" event={"ID":"dad1d93c-7486-4e99-8a28-108dbf4b659d","Type":"ContainerDied","Data":"aa6fbfd346067de0ebac86051f90cbd1cf42572bc21e13c0cdf7225702e99dd6"}
Oct 30 21:26:16.468629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:16.466965    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:16.468629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:16.467045    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:17.142627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.122888    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.201637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.188273    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.201637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.193872    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.271627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.249262    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:17.429697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.412347    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.464159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.462759    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.471232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.470902    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:17.471232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:17.470974    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:18.138688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.126132    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:26:18.173660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.169206    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:26:18.199669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.195193    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:18.199669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.196125    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:26:18.263928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.256286    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:18.263928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.257052    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:26:18.454639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.441263    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:26:18.480631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.455541    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:26:18.552699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.482781    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:18.561642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.510271    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:18.561902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.561863    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:18.565799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:18.565765    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:26:19.184903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.151659    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.199190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.199140    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.200021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.199985    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.364870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.364806    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:19.378739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.378688    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.443888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.435757    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:19.472671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.468652    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:19.472671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.470045    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.501846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.501791    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:26:19.514396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.514356    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:19.689130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.617755    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:19.689130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.687646    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:19.773740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.756362    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:19.810283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.810244    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:26:19.998624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.969200    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca\") pod \"dad1d93c-7486-4e99-8a28-108dbf4b659d\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") "
Oct 30 21:26:19.998624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.976688    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq\" (UniqueName: \"kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq\") pod \"dad1d93c-7486-4e99-8a28-108dbf4b659d\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") "
Oct 30 21:26:19.998624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.976766    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca\") pod \"dad1d93c-7486-4e99-8a28-108dbf4b659d\" (UID: \"dad1d93c-7486-4e99-8a28-108dbf4b659d\") "
Oct 30 21:26:19.998624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:19.977859    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca" pod "dad1d93c-7486-4e99-8a28-108dbf4b659d" (UID: "dad1d93c-7486-4e99-8a28-108dbf4b659d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:20.111968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.103868    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:20.163928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.133762    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf" event={"ID":"dad1d93c-7486-4e99-8a28-108dbf4b659d","Type":"ContainerDied","Data":"48b608612a4e2b436454a469dcb344ca1f86fb04bacc1aac15fee94c203579c9"}
Oct 30 21:26:20.164455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.164431    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b608612a4e2b436454a469dcb344ca1f86fb04bacc1aac15fee94c203579c9"
Oct 30 21:26:20.164539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.163866    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:20.164972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.133965    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"
Oct 30 21:26:20.169117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.169056    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca" pod "dad1d93c-7486-4e99-8a28-108dbf4b659d" (UID: "dad1d93c-7486-4e99-8a28-108dbf4b659d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:20.234627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.227065    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:20.287627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.286944    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k" event={"ID":"4f4b0601-8bf6-41d8-a382-a70121071475","Type":"ContainerDied","Data":"b5a0a61ac052c70703c8d29ae7124559cfbda5dfc9c9e4c1ace3ae97b6d2a621"}
Oct 30 21:26:20.287627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.287011    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5a0a61ac052c70703c8d29ae7124559cfbda5dfc9c9e4c1ace3ae97b6d2a621"
Oct 30 21:26:20.293289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.293256    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:26:20.313610 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.313568    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:26:20.314912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.295964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca\") pod \"4f4b0601-8bf6-41d8-a382-a70121071475\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") "
Oct 30 21:26:20.315162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.315128    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca\") pod \"4f4b0601-8bf6-41d8-a382-a70121071475\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") "
Oct 30 21:26:20.368115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.296368    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca" pod "4f4b0601-8bf6-41d8-a382-a70121071475" (UID: "4f4b0601-8bf6-41d8-a382-a70121071475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:20.372691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.372657    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:26:20.387110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.380910    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9\" (UniqueName: \"kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9\") pod \"4f4b0601-8bf6-41d8-a382-a70121071475\" (UID: \"4f4b0601-8bf6-41d8-a382-a70121071475\") "
Oct 30 21:26:20.387110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.381805    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad1d93c-7486-4e99-8a28-108dbf4b659d-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:20.387110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.381830    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:20.399888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.396917    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq" pod "dad1d93c-7486-4e99-8a28-108dbf4b659d" (UID: "dad1d93c-7486-4e99-8a28-108dbf4b659d"). InnerVolumeSpecName "kube-api-access-6s9pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:20.399888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.398358    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:26:20.399888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.398691    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca" pod "4f4b0601-8bf6-41d8-a382-a70121071475" (UID: "4f4b0601-8bf6-41d8-a382-a70121071475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:20.409002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.408964    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9" pod "4f4b0601-8bf6-41d8-a382-a70121071475" (UID: "4f4b0601-8bf6-41d8-a382-a70121071475"). InnerVolumeSpecName "kube-api-access-qzwt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:20.466802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.466761    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:20.466950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.466825    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:20.483353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.483139    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwt9\" (UniqueName: \"kubernetes.io/projected/4f4b0601-8bf6-41d8-a382-a70121071475-kube-api-access-qzwt9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:20.483353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.483176    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4b0601-8bf6-41d8-a382-a70121071475-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:20.483353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:20.483191    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9pq\" (UniqueName: \"kubernetes.io/projected/dad1d93c-7486-4e99-8a28-108dbf4b659d-kube-api-access-6s9pq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:21.276751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.276721    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"
Oct 30 21:26:21.466018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.465981    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:21.466172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.466037    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:21.493203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493166    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"]
Oct 30 21:26:21.493871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493850    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a98c7fc3-b122-4bff-bb9d-1c8471f0223e" containerName="revoke-keys"
Oct 30 21:26:21.493871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493871    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a98c7fc3-b122-4bff-bb9d-1c8471f0223e" containerName="revoke-keys"
Oct 30 21:26:21.494026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493902    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4f4b0601-8bf6-41d8-a382-a70121071475" containerName="revoke-keys"
Oct 30 21:26:21.494026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493909    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4b0601-8bf6-41d8-a382-a70121071475" containerName="revoke-keys"
Oct 30 21:26:21.494026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493953    6695 cpu_manager.go:414] "containerMap: removing container" podUID="dad1d93c-7486-4e99-8a28-108dbf4b659d" containerName="revoke-keys"
Oct 30 21:26:21.494026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.493961    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad1d93c-7486-4e99-8a28-108dbf4b659d" containerName="revoke-keys"
Oct 30 21:26:21.494236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.494086    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a98c7fc3-b122-4bff-bb9d-1c8471f0223e" containerName="revoke-keys"
Oct 30 21:26:21.494236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.494099    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4b0601-8bf6-41d8-a382-a70121071475" containerName="revoke-keys"
Oct 30 21:26:21.494236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.494118    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad1d93c-7486-4e99-8a28-108dbf4b659d" containerName="revoke-keys"
Oct 30 21:26:21.498043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.498014    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.537627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.529943    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:26:21.537627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.530071    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:21.537627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.530154    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:21.537627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.530780    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:21.556319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.554327    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"]
Oct 30 21:26:21.605291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.605249    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkzg\" (UniqueName: \"kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.605506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.605404    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.605506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.605451    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.643801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.643759    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"]
Oct 30 21:26:21.648166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.648143    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.663513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.663484    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"]
Oct 30 21:26:21.706984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.706946    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.707136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.707000    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.707136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.707094    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.707235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.707150    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.707310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.707239    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmfx\" (UniqueName: \"kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.707360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.707341    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkzg\" (UniqueName: \"kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.707588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.707567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.709367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.709348    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.759629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.759540    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkzg\" (UniqueName: \"kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg\") pod \"maas-api-revoke-sub-1dcaa342-pmdd5\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.811680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.809094    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.811680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.809215    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.811680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.809251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmfx\" (UniqueName: \"kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.826630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.821857    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.829617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.829533    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.836144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.831865    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:21.848804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.839569    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmfx\" (UniqueName: \"kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx\") pod \"maas-api-revoke-sub-9de0cb0f-r5hpt\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:21.964654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:21.964312    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:22.201671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.201555    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"]
Oct 30 21:26:22.205571 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:22.205534    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb758a0aa_f78e_490b_8069_71758a4ffdd0.slice/crio-403f8a5753cd247a888304e566ee585bb0166baaa32019d108190e1c22806b63 WatchSource:0}: Error finding container 403f8a5753cd247a888304e566ee585bb0166baaa32019d108190e1c22806b63: Status 404 returned error can't find the container with id 403f8a5753cd247a888304e566ee585bb0166baaa32019d108190e1c22806b63
Oct 30 21:26:22.285567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.285532    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5" event={"ID":"b758a0aa-f78e-490b-8069-71758a4ffdd0","Type":"ContainerStarted","Data":"403f8a5753cd247a888304e566ee585bb0166baaa32019d108190e1c22806b63"}
Oct 30 21:26:22.305991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.305867    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"]
Oct 30 21:26:22.308773 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:22.308740    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4f7d23_1179_459c_a97c_5ae1420f0b85.slice/crio-5f5f69da604a7f43cd708737ca14cf303f155e97b3160a561f44382b4c9f48c0 WatchSource:0}: Error finding container 5f5f69da604a7f43cd708737ca14cf303f155e97b3160a561f44382b4c9f48c0: Status 404 returned error can't find the container with id 5f5f69da604a7f43cd708737ca14cf303f155e97b3160a561f44382b4c9f48c0
Oct 30 21:26:22.466476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.466442    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:22.466659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.466492    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:22.592530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.592449    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"]
Oct 30 21:26:22.598133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.598106    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.655068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.653006    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"]
Oct 30 21:26:22.757093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.757058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.757277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.757223    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.757337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.757283    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.858342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.858244    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.858519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.858360    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.858519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.858472    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.859116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.859095    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.860892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.860866    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.876764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.876740    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr\") pod \"maas-api-revoke-sub-e31b8df8-vf7w5\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:22.918445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:22.918412    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:23.232117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.232077    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"]
Oct 30 21:26:23.297193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.297159    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5" event={"ID":"c3fab7bd-49b9-41e5-ba47-d406e826074e","Type":"ContainerStarted","Data":"8e60734d3383cb9870eff8d935203be068383622977c9bfc6142eeecd388f23b"}
Oct 30 21:26:23.299153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.299123    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5" event={"ID":"b758a0aa-f78e-490b-8069-71758a4ffdd0","Type":"ContainerStarted","Data":"d38b92c52e2e695e49face1fa3f217f328cb654099fc03a5064f7634ff3f166d"}
Oct 30 21:26:23.301355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.301324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt" event={"ID":"bd4f7d23-1179-459c-a97c-5ae1420f0b85","Type":"ContainerStarted","Data":"b0e06407607516503b71be5c3a4af30198bd85a24325e019e16096b9914592d0"}
Oct 30 21:26:23.301485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.301380    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt" event={"ID":"bd4f7d23-1179-459c-a97c-5ae1420f0b85","Type":"ContainerStarted","Data":"5f5f69da604a7f43cd708737ca14cf303f155e97b3160a561f44382b4c9f48c0"}
Oct 30 21:26:23.337023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.336695    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5" podStartSLOduration=2.336680641 podStartE2EDuration="2.336680641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:23.335966777 +0000 UTC m=+966.258425076" watchObservedRunningTime="2026-10-30 21:26:23.336680641 +0000 UTC m=+966.259138938"
Oct 30 21:26:23.376377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.376320    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt" podStartSLOduration=2.376305868 podStartE2EDuration="2.376305868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:23.371476655 +0000 UTC m=+966.293934956" watchObservedRunningTime="2026-10-30 21:26:23.376305868 +0000 UTC m=+966.298764163"
Oct 30 21:26:23.466496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.466450    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:26:23.466687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:23.466522    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:26:24.311722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.311671    6695 generic.go:356] "Generic (PLEG): container finished" podID="c3fab7bd-49b9-41e5-ba47-d406e826074e" containerID="70c632da9c99767c5f843eccbf0e1ffb8b898c423943d0ea1a08eff8700fef2f" exitCode=0
Oct 30 21:26:24.312180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.311859    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5" event={"ID":"c3fab7bd-49b9-41e5-ba47-d406e826074e","Type":"ContainerDied","Data":"70c632da9c99767c5f843eccbf0e1ffb8b898c423943d0ea1a08eff8700fef2f"}
Oct 30 21:26:24.314042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.314011    6695 generic.go:356] "Generic (PLEG): container finished" podID="b758a0aa-f78e-490b-8069-71758a4ffdd0" containerID="d38b92c52e2e695e49face1fa3f217f328cb654099fc03a5064f7634ff3f166d" exitCode=0
Oct 30 21:26:24.314225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.314173    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5" event={"ID":"b758a0aa-f78e-490b-8069-71758a4ffdd0","Type":"ContainerDied","Data":"d38b92c52e2e695e49face1fa3f217f328cb654099fc03a5064f7634ff3f166d"}
Oct 30 21:26:24.316281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.316253    6695 generic.go:356] "Generic (PLEG): container finished" podID="bd4f7d23-1179-459c-a97c-5ae1420f0b85" containerID="b0e06407607516503b71be5c3a4af30198bd85a24325e019e16096b9914592d0" exitCode=0
Oct 30 21:26:24.316422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.316356    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt" event={"ID":"bd4f7d23-1179-459c-a97c-5ae1420f0b85","Type":"ContainerDied","Data":"b0e06407607516503b71be5c3a4af30198bd85a24325e019e16096b9914592d0"}
Oct 30 21:26:24.899939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:24.899909    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:25.026438 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:25.026403    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:25.026438 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:25.026436    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d4877f77-k9hwm: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:25.026864 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:25.026844    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials podName:12ef6d10-d973-4031-9f74-80a6d305f576 nodeName:}" failed. No retries permitted until 2026-10-30 21:26:25.526817778 +0000 UTC m=+968.449276070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials") pod "payload-processing-external-model-d4877f77-k9hwm" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576") : secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:25.187907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.187744    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"]
Oct 30 21:26:25.469110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.466901    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:25.469110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.466971    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.480740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.480708    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:25.481686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.481620    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:25.488276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.488254    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:26:25.565240 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:25.548598    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:25.565240 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:25.548647    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d4877f77-k9hwm: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:25.565240 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:25.548709    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials podName:12ef6d10-d973-4031-9f74-80a6d305f576 nodeName:}" failed. No retries permitted until 2026-10-30 21:26:26.548688592 +0000 UTC m=+969.471146868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials") pod "payload-processing-external-model-d4877f77-k9hwm" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576") : secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:25.774689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.771804    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:25.778796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.778756    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx\" (UniqueName: \"kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx\") pod \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") "
Oct 30 21:26:25.780357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.780298    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca" pod "bd4f7d23-1179-459c-a97c-5ae1420f0b85" (UID: "bd4f7d23-1179-459c-a97c-5ae1420f0b85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:25.780789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.780767    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca\") pod \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") "
Oct 30 21:26:25.781051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.781032    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca\") pod \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\" (UID: \"bd4f7d23-1179-459c-a97c-5ae1420f0b85\") "
Oct 30 21:26:25.781970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.781949    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:25.790172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.787965    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx" pod "bd4f7d23-1179-459c-a97c-5ae1420f0b85" (UID: "bd4f7d23-1179-459c-a97c-5ae1420f0b85"). InnerVolumeSpecName "kube-api-access-zgmfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:25.790172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.788102    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca" pod "bd4f7d23-1179-459c-a97c-5ae1420f0b85" (UID: "bd4f7d23-1179-459c-a97c-5ae1420f0b85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:25.898167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.898075    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgmfx\" (UniqueName: \"kubernetes.io/projected/bd4f7d23-1179-459c-a97c-5ae1420f0b85-kube-api-access-zgmfx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:25.898167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.898114    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd4f7d23-1179-459c-a97c-5ae1420f0b85-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:25.923641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.922432    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:25.943588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.943557    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:25.999424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999385    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca\") pod \"c3fab7bd-49b9-41e5-ba47-d406e826074e\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") "
Oct 30 21:26:25.999639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999457    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg\" (UniqueName: \"kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg\") pod \"b758a0aa-f78e-490b-8069-71758a4ffdd0\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") "
Oct 30 21:26:25.999639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999515    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr\") pod \"c3fab7bd-49b9-41e5-ba47-d406e826074e\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") "
Oct 30 21:26:25.999639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999551    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca\") pod \"b758a0aa-f78e-490b-8069-71758a4ffdd0\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") "
Oct 30 21:26:25.999840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999746    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca\") pod \"b758a0aa-f78e-490b-8069-71758a4ffdd0\" (UID: \"b758a0aa-f78e-490b-8069-71758a4ffdd0\") "
Oct 30 21:26:25.999840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999797    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca\") pod \"c3fab7bd-49b9-41e5-ba47-d406e826074e\" (UID: \"c3fab7bd-49b9-41e5-ba47-d406e826074e\") "
Oct 30 21:26:25.999950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:25.999837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca" pod "c3fab7bd-49b9-41e5-ba47-d406e826074e" (UID: "c3fab7bd-49b9-41e5-ba47-d406e826074e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:26.000235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.000203    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca" pod "b758a0aa-f78e-490b-8069-71758a4ffdd0" (UID: "b758a0aa-f78e-490b-8069-71758a4ffdd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:26.000543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.000521    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:26.000670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.000550    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:26.002100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.002077    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg" pod "b758a0aa-f78e-490b-8069-71758a4ffdd0" (UID: "b758a0aa-f78e-490b-8069-71758a4ffdd0"). InnerVolumeSpecName "kube-api-access-5dkzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:26.002269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.002250    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca" pod "b758a0aa-f78e-490b-8069-71758a4ffdd0" (UID: "b758a0aa-f78e-490b-8069-71758a4ffdd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:26.002269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.002262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr" pod "c3fab7bd-49b9-41e5-ba47-d406e826074e" (UID: "c3fab7bd-49b9-41e5-ba47-d406e826074e"). InnerVolumeSpecName "kube-api-access-pljbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:26.002503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.002486    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca" pod "c3fab7bd-49b9-41e5-ba47-d406e826074e" (UID: "c3fab7bd-49b9-41e5-ba47-d406e826074e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:26.102229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.102144    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3fab7bd-49b9-41e5-ba47-d406e826074e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:26.102229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.102177    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dkzg\" (UniqueName: \"kubernetes.io/projected/b758a0aa-f78e-490b-8069-71758a4ffdd0-kube-api-access-5dkzg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:26.102229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.102188    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/c3fab7bd-49b9-41e5-ba47-d406e826074e-kube-api-access-pljbr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:26.102229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.102198    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b758a0aa-f78e-490b-8069-71758a4ffdd0-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:26.336345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.336315    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"
Oct 30 21:26:26.336519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.336352    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5" event={"ID":"b758a0aa-f78e-490b-8069-71758a4ffdd0","Type":"ContainerDied","Data":"403f8a5753cd247a888304e566ee585bb0166baaa32019d108190e1c22806b63"}
Oct 30 21:26:26.336519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.336398    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="403f8a5753cd247a888304e566ee585bb0166baaa32019d108190e1c22806b63"
Oct 30 21:26:26.337919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.337901    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"
Oct 30 21:26:26.338050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.337940    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt" event={"ID":"bd4f7d23-1179-459c-a97c-5ae1420f0b85","Type":"ContainerDied","Data":"5f5f69da604a7f43cd708737ca14cf303f155e97b3160a561f44382b4c9f48c0"}
Oct 30 21:26:26.338050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.337975    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f5f69da604a7f43cd708737ca14cf303f155e97b3160a561f44382b4c9f48c0"
Oct 30 21:26:26.339518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.339497    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"
Oct 30 21:26:26.339641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.339533    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5" event={"ID":"c3fab7bd-49b9-41e5-ba47-d406e826074e","Type":"ContainerDied","Data":"8e60734d3383cb9870eff8d935203be068383622977c9bfc6142eeecd388f23b"}
Oct 30 21:26:26.339641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:26.339562    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e60734d3383cb9870eff8d935203be068383622977c9bfc6142eeecd388f23b"
Oct 30 21:26:26.610104 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:26.610070    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:26.610104 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:26.610102    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d4877f77-k9hwm: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:26.610558 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:26.610164    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials podName:12ef6d10-d973-4031-9f74-80a6d305f576 nodeName:}" failed. No retries permitted until 2026-10-30 21:26:28.610145565 +0000 UTC m=+971.532603860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials") pod "payload-processing-external-model-d4877f77-k9hwm" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576") : secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:28.644032 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:28.643993    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9d93cf183177: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:28.644032 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:28.644024    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d4877f77-k9hwm: secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:28.644495 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:28.644075    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials podName:12ef6d10-d973-4031-9f74-80a6d305f576 nodeName:}" failed. No retries permitted until 2026-10-30 21:26:32.644060944 +0000 UTC m=+975.566519233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials") pod "payload-processing-external-model-d4877f77-k9hwm" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576") : secret "agc-cred-9d93cf183177" not found
Oct 30 21:26:28.872816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.872762    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"]
Oct 30 21:26:28.873515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873494    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c3fab7bd-49b9-41e5-ba47-d406e826074e" containerName="revoke-keys"
Oct 30 21:26:28.873571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873517    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3fab7bd-49b9-41e5-ba47-d406e826074e" containerName="revoke-keys"
Oct 30 21:26:28.873571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873533    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b758a0aa-f78e-490b-8069-71758a4ffdd0" containerName="revoke-keys"
Oct 30 21:26:28.873571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873538    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b758a0aa-f78e-490b-8069-71758a4ffdd0" containerName="revoke-keys"
Oct 30 21:26:28.873571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873553    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bd4f7d23-1179-459c-a97c-5ae1420f0b85" containerName="revoke-keys"
Oct 30 21:26:28.873571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873558    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4f7d23-1179-459c-a97c-5ae1420f0b85" containerName="revoke-keys"
Oct 30 21:26:28.873772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873682    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4f7d23-1179-459c-a97c-5ae1420f0b85" containerName="revoke-keys"
Oct 30 21:26:28.873772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873698    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3fab7bd-49b9-41e5-ba47-d406e826074e" containerName="revoke-keys"
Oct 30 21:26:28.873772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.873705    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b758a0aa-f78e-490b-8069-71758a4ffdd0" containerName="revoke-keys"
Oct 30 21:26:28.877691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.877671    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:28.893140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.893112    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"]
Oct 30 21:26:28.901701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.899906    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:28.901701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.899906    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:28.948492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.948454    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzlpp\" (UniqueName: \"kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:28.948690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.948511    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:28.948690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.948622    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:28.951171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.951152    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:28.951219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:28.951202    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:26:29.050488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.050448    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzlpp\" (UniqueName: \"kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.050714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.050510    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.050714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.050693    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.051598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.051574    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.053079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.053058    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.069217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.069193    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzlpp\" (UniqueName: \"kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp\") pod \"maas-api-revoke-sub-925dad17-zmsgz\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.199042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.194913    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:29.512704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:29.512670    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"]
Oct 30 21:26:29.515236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:29.515202    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5b6f9b_3973_4d62_b9c0_1506672b7927.slice/crio-dba7bbf13c04bda24e408f514b88ec4a8ad922235f2c0cb96b9de7b51dc0f94b WatchSource:0}: Error finding container dba7bbf13c04bda24e408f514b88ec4a8ad922235f2c0cb96b9de7b51dc0f94b: Status 404 returned error can't find the container with id dba7bbf13c04bda24e408f514b88ec4a8ad922235f2c0cb96b9de7b51dc0f94b
Oct 30 21:26:30.329872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.329799    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm" podUID="12ef6d10-d973-4031-9f74-80a6d305f576" containerName="payload-processing" containerID="cri-o://af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137" gracePeriod=25
Oct 30 21:26:30.380528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.380488    6695 generic.go:356] "Generic (PLEG): container finished" podID="ad5b6f9b-3973-4d62-b9c0-1506672b7927" containerID="29432fa378b18d20eb63fa97011e067726d96af633d3e63e7d1dace4965ec850" exitCode=0
Oct 30 21:26:30.380731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.380574    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz" event={"ID":"ad5b6f9b-3973-4d62-b9c0-1506672b7927","Type":"ContainerDied","Data":"29432fa378b18d20eb63fa97011e067726d96af633d3e63e7d1dace4965ec850"}
Oct 30 21:26:30.380731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.380679    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz" event={"ID":"ad5b6f9b-3973-4d62-b9c0-1506672b7927","Type":"ContainerStarted","Data":"dba7bbf13c04bda24e408f514b88ec4a8ad922235f2c0cb96b9de7b51dc0f94b"}
Oct 30 21:26:30.614827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.614797    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:30.675754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.675720    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p\") pod \"12ef6d10-d973-4031-9f74-80a6d305f576\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") "
Oct 30 21:26:30.675997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.675803    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume\") pod \"12ef6d10-d973-4031-9f74-80a6d305f576\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") "
Oct 30 21:26:30.675997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.675913    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume\") pod \"12ef6d10-d973-4031-9f74-80a6d305f576\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") "
Oct 30 21:26:30.676142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.676100    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials\") pod \"12ef6d10-d973-4031-9f74-80a6d305f576\" (UID: \"12ef6d10-d973-4031-9f74-80a6d305f576\") "
Oct 30 21:26:30.676291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.676262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume" pod "12ef6d10-d973-4031-9f74-80a6d305f576" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:30.676748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.676724    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:30.676748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.676677    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume" pod "12ef6d10-d973-4031-9f74-80a6d305f576" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:30.678583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.678555    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p" pod "12ef6d10-d973-4031-9f74-80a6d305f576" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576"). InnerVolumeSpecName "kube-api-access-qrf9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:30.678690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.678578    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials" pod "12ef6d10-d973-4031-9f74-80a6d305f576" (UID: "12ef6d10-d973-4031-9f74-80a6d305f576"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:30.780066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.780020    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrf9p\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-kube-api-access-qrf9p\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:30.780066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.780066    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ef6d10-d973-4031-9f74-80a6d305f576-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:30.780384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:30.780081    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ef6d10-d973-4031-9f74-80a6d305f576-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:31.074835 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:31.074795    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:26:31.389928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.389842    6695 generic.go:356] "Generic (PLEG): container finished" podID="12ef6d10-d973-4031-9f74-80a6d305f576" containerID="af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137" exitCode=0
Oct 30 21:26:31.389928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.389891    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"
Oct 30 21:26:31.390386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.389939    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm" event={"ID":"12ef6d10-d973-4031-9f74-80a6d305f576","Type":"ContainerDied","Data":"af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137"}
Oct 30 21:26:31.390386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.390002    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d4877f77-k9hwm" event={"ID":"12ef6d10-d973-4031-9f74-80a6d305f576","Type":"ContainerDied","Data":"f5c7b5945e90a1099161013e8601b7f598d00597564284dffdf710d949f00e18"}
Oct 30 21:26:31.390386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.390033    6695 scope.go:122] "RemoveContainer" containerID="af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137"
Oct 30 21:26:31.404035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.404010    6695 scope.go:122] "RemoveContainer" containerID="af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137"
Oct 30 21:26:31.404428 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:26:31.404400    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137\": container with ID starting with af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137 not found: ID does not exist" containerID="af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137"
Oct 30 21:26:31.404518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.404438    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137"} err="failed to get container status \"af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137\": rpc error: code = NotFound desc = could not find container \"af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137\": container with ID starting with af21556965d148836d7cd1da70d662d93907c6317ba878e1787a5d3208f17137 not found: ID does not exist"
Oct 30 21:26:31.442576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.442543    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"]
Oct 30 21:26:31.456031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.455829    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d4877f77-k9hwm"]
Oct 30 21:26:31.565327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.565300    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:31.691987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.691896    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca\") pod \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") "
Oct 30 21:26:31.691987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.691966    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp\" (UniqueName: \"kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp\") pod \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") "
Oct 30 21:26:31.692203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.692111    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca\") pod \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\" (UID: \"ad5b6f9b-3973-4d62-b9c0-1506672b7927\") "
Oct 30 21:26:31.692314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.692265    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca" pod "ad5b6f9b-3973-4d62-b9c0-1506672b7927" (UID: "ad5b6f9b-3973-4d62-b9c0-1506672b7927"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:31.692748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.692723    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:31.697960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.694262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca" pod "ad5b6f9b-3973-4d62-b9c0-1506672b7927" (UID: "ad5b6f9b-3973-4d62-b9c0-1506672b7927"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:31.697960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.694366    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp" pod "ad5b6f9b-3973-4d62-b9c0-1506672b7927" (UID: "ad5b6f9b-3973-4d62-b9c0-1506672b7927"). InnerVolumeSpecName "kube-api-access-nzlpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:31.794392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.794349    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzlpp\" (UniqueName: \"kubernetes.io/projected/ad5b6f9b-3973-4d62-b9c0-1506672b7927-kube-api-access-nzlpp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:31.794392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:31.794395    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5b6f9b-3973-4d62-b9c0-1506672b7927-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:32.466631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:32.464729    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz" event={"ID":"ad5b6f9b-3973-4d62-b9c0-1506672b7927","Type":"ContainerDied","Data":"dba7bbf13c04bda24e408f514b88ec4a8ad922235f2c0cb96b9de7b51dc0f94b"}
Oct 30 21:26:32.466631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:32.464807    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba7bbf13c04bda24e408f514b88ec4a8ad922235f2c0cb96b9de7b51dc0f94b"
Oct 30 21:26:32.466631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:32.464897    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"
Oct 30 21:26:33.297633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.291652    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ef6d10-d973-4031-9f74-80a6d305f576" path="/var/lib/kubelet/pods/12ef6d10-d973-4031-9f74-80a6d305f576/volumes"
Oct 30 21:26:33.309677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.309596    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:33.361537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.361495    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:33.498079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.498025    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:33.705681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.701749    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:26:33.731912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.731051    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.736577    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"]
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.737474    6695 cpu_manager.go:414] "containerMap: removing container" podUID="12ef6d10-d973-4031-9f74-80a6d305f576" containerName="payload-processing"
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.737495    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ef6d10-d973-4031-9f74-80a6d305f576" containerName="payload-processing"
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.737571    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ad5b6f9b-3973-4d62-b9c0-1506672b7927" containerName="revoke-keys"
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.737580    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5b6f9b-3973-4d62-b9c0-1506672b7927" containerName="revoke-keys"
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.737799    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5b6f9b-3973-4d62-b9c0-1506672b7927" containerName="revoke-keys"
Oct 30 21:26:33.755622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.737817    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ef6d10-d973-4031-9f74-80a6d305f576" containerName="payload-processing"
Oct 30 21:26:33.761283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.761249    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.779370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.778502    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:33.779370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.779228    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:26:33.779370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.779303    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:33.792630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.791922    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:33.820021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.819977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.820346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.820319    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.820497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.820477    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zml\" (UniqueName: \"kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.847929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.847893    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"]
Oct 30 21:26:33.927624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.925797    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.927624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.925859    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zml\" (UniqueName: \"kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.927624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.926163    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.927924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.927706    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.969678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.961469    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zml\" (UniqueName: \"kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:33.991657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:33.991112    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe10e0c0-9bqrd\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:34.154877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.154836    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:34.156563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.156522    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.194961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.194874    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.203728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.203682    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:34.352709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.348572    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.365641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.353510    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:26:34.770064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.770014    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.770535    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.780868    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.780936    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.781018    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.781415    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.828894    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:34.927751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:34.828988    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.367131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.365890    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:35.521992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.500683    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:35.559634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.522961    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.559634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.523581    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.565851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.565806    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9" podUID="583d659a-c61c-4378-a025-f370c0dd1472" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.675124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.620483    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:26:35.684772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.684717    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.801630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.800174    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.834744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.834690    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:36.030690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.990093    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:36.030690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.990160    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:36.030690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:35.977989    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:26:36.030690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.004098    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:26:36.030690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.004330    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:26:36.388224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.377671    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:36.505629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.501698    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:36.534134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.534096    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:26:36.705687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.693378    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:36.828636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.808925    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:36.869628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:36.861708    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.255311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.247000    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-k86d4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:37.255311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.247084    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-k86d4" podUID="ec504004-13c4-4d32-882e-a60f4b4d1db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.372494 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:37.371111    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4813512f_df13_40c1_aa89_c1f9d07840d7.slice/crio-a41033bd939c270cd7902c05f94767520f3a8d65c42f58de0fb44ab507210de3 WatchSource:0}: Error finding container a41033bd939c270cd7902c05f94767520f3a8d65c42f58de0fb44ab507210de3: Status 404 returned error can't find the container with id a41033bd939c270cd7902c05f94767520f3a8d65c42f58de0fb44ab507210de3
Oct 30 21:26:37.395748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.395697    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.560691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.515730    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:37.560691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.537794    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.800629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.758902    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:37.800629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.758975    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.800629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.786203    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.816711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.801467    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:37.816711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.801541    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.854630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.818176    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:26:37.879107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:37.879055    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:38.178634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.168782    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd" event={"ID":"4813512f-df13-40c1-aa89-c1f9d07840d7","Type":"ContainerStarted","Data":"a41033bd939c270cd7902c05f94767520f3a8d65c42f58de0fb44ab507210de3"}
Oct 30 21:26:38.187922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.185327    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"]
Oct 30 21:26:38.401234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.401119    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:38.434576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.422445    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:38.434576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.422528    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:38.527742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.527687    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:38.543825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.543715    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:38.757686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.749499    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:26:38.790844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.790791    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:38.904228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:38.904185    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:39.329638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:39.324384    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:26:39.335512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:39.330463    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:26:39.490769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:39.490540    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:26:39.555973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:39.555946    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:26:39.577302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:39.544430    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:39.619580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:39.619552    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:26:40.439671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.439576    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"]
Oct 30 21:26:40.444047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.444022    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.462484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.462443    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"]
Oct 30 21:26:40.559946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.559913    6695 generic.go:356] "Generic (PLEG): container finished" podID="4813512f-df13-40c1-aa89-c1f9d07840d7" containerID="22c1ca597bf2cdbd2f88cbb781ebc98701598e2137a1c79ac0b5df9c1b026f09" exitCode=0
Oct 30 21:26:40.560101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.559994    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd" event={"ID":"4813512f-df13-40c1-aa89-c1f9d07840d7","Type":"ContainerDied","Data":"22c1ca597bf2cdbd2f88cbb781ebc98701598e2137a1c79ac0b5df9c1b026f09"}
Oct 30 21:26:40.563869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.563812    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g2bk\" (UniqueName: \"kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.564058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.563943    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.564058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.564035    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.665008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.664973    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g2bk\" (UniqueName: \"kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.665184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.665042    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.665184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.665124    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.665924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.665894    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.668381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.668354    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.684197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.684167    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g2bk\" (UniqueName: \"kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk\") pod \"maas-api-revoke-sub-55abbb3e-9t5qr\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:40.764052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:40.763977    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:41.079729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.079695    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"]
Oct 30 21:26:41.085578 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:41.085546    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6fdd92_d7a6_4b6e_85c5_04a94152335b.slice/crio-97e9e693e21810c23784362d24c22c9c9503bbdad9a2e44e2d6d6f175adc780d WatchSource:0}: Error finding container 97e9e693e21810c23784362d24c22c9c9503bbdad9a2e44e2d6d6f175adc780d: Status 404 returned error can't find the container with id 97e9e693e21810c23784362d24c22c9c9503bbdad9a2e44e2d6d6f175adc780d
Oct 30 21:26:41.570450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.570386    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr" event={"ID":"4d6fdd92-d7a6-4b6e-85c5-04a94152335b","Type":"ContainerStarted","Data":"571447f1ce42db87d640a9f8d9bd22cb0c26d6ceb7f6c395f369467f9e453c12"}
Oct 30 21:26:41.570917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.570745    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr" event={"ID":"4d6fdd92-d7a6-4b6e-85c5-04a94152335b","Type":"ContainerStarted","Data":"97e9e693e21810c23784362d24c22c9c9503bbdad9a2e44e2d6d6f175adc780d"}
Oct 30 21:26:41.614580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.614476    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr" podStartSLOduration=1.614461569 podStartE2EDuration="1.614461569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:41.610760161 +0000 UTC m=+984.533218459" watchObservedRunningTime="2026-10-30 21:26:41.614461569 +0000 UTC m=+984.536919866"
Oct 30 21:26:41.740744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.740713    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:41.882294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.882200    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca\") pod \"4813512f-df13-40c1-aa89-c1f9d07840d7\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") "
Oct 30 21:26:41.882459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.882344    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca\") pod \"4813512f-df13-40c1-aa89-c1f9d07840d7\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") "
Oct 30 21:26:41.882459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.882397    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml\" (UniqueName: \"kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml\") pod \"4813512f-df13-40c1-aa89-c1f9d07840d7\" (UID: \"4813512f-df13-40c1-aa89-c1f9d07840d7\") "
Oct 30 21:26:41.882616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.882574    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca" pod "4813512f-df13-40c1-aa89-c1f9d07840d7" (UID: "4813512f-df13-40c1-aa89-c1f9d07840d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:41.882934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.882916    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:41.884492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.884467    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca" pod "4813512f-df13-40c1-aa89-c1f9d07840d7" (UID: "4813512f-df13-40c1-aa89-c1f9d07840d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:41.884729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.884708    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml" pod "4813512f-df13-40c1-aa89-c1f9d07840d7" (UID: "4813512f-df13-40c1-aa89-c1f9d07840d7"). InnerVolumeSpecName "kube-api-access-v9zml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:41.984302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.984252    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zml\" (UniqueName: \"kubernetes.io/projected/4813512f-df13-40c1-aa89-c1f9d07840d7-kube-api-access-v9zml\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:41.984302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:41.984303    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4813512f-df13-40c1-aa89-c1f9d07840d7-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:42.579876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:42.579844    6695 generic.go:356] "Generic (PLEG): container finished" podID="4d6fdd92-d7a6-4b6e-85c5-04a94152335b" containerID="571447f1ce42db87d640a9f8d9bd22cb0c26d6ceb7f6c395f369467f9e453c12" exitCode=0
Oct 30 21:26:42.580357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:42.579932    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr" event={"ID":"4d6fdd92-d7a6-4b6e-85c5-04a94152335b","Type":"ContainerDied","Data":"571447f1ce42db87d640a9f8d9bd22cb0c26d6ceb7f6c395f369467f9e453c12"}
Oct 30 21:26:42.581715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:42.581692    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"
Oct 30 21:26:42.581844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:42.581712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd" event={"ID":"4813512f-df13-40c1-aa89-c1f9d07840d7","Type":"ContainerDied","Data":"a41033bd939c270cd7902c05f94767520f3a8d65c42f58de0fb44ab507210de3"}
Oct 30 21:26:42.581844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:42.581752    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41033bd939c270cd7902c05f94767520f3a8d65c42f58de0fb44ab507210de3"
Oct 30 21:26:43.227535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.227499    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"]
Oct 30 21:26:43.228158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.228138    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4813512f-df13-40c1-aa89-c1f9d07840d7" containerName="revoke-keys"
Oct 30 21:26:43.228257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.228159    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4813512f-df13-40c1-aa89-c1f9d07840d7" containerName="revoke-keys"
Oct 30 21:26:43.228366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.228352    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4813512f-df13-40c1-aa89-c1f9d07840d7" containerName="revoke-keys"
Oct 30 21:26:43.232027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.232006    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.259997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.259964    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"]
Oct 30 21:26:43.311268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311230    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311311    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311424    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311523    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwnn\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311649    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311692    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311727    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311791    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.311912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.311895    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413176    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413234    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413286    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwnn\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413318    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413359    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413401    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413417    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413455    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413546    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413682    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413701    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413804    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.413899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.413833    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.414251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.414230    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.415980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.415955    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.416096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.416079    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.431901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.431869    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.432034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.431955    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwnn\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.552343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.552310    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:43.763039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.763010    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:43.887121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.887038    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"]
Oct 30 21:26:43.890662 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:43.890632    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b64f18b_9634_486a_9ec2_a52afa35fd99.slice/crio-1a111b34f2c17d5a1fc26b2945fce53f27d983418757d1406e650f3a76e25932 WatchSource:0}: Error finding container 1a111b34f2c17d5a1fc26b2945fce53f27d983418757d1406e650f3a76e25932: Status 404 returned error can't find the container with id 1a111b34f2c17d5a1fc26b2945fce53f27d983418757d1406e650f3a76e25932
Oct 30 21:26:43.893238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.893203    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:43.893334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.893269    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:43.893334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.893294    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:43.921046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.921014    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk\" (UniqueName: \"kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk\") pod \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") "
Oct 30 21:26:43.921172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.921145    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca\") pod \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") "
Oct 30 21:26:43.921298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.921280    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca\") pod \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\" (UID: \"4d6fdd92-d7a6-4b6e-85c5-04a94152335b\") "
Oct 30 21:26:43.921618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.921577    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca" pod "4d6fdd92-d7a6-4b6e-85c5-04a94152335b" (UID: "4d6fdd92-d7a6-4b6e-85c5-04a94152335b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:43.921869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.921850    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:43.923070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.923049    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk" pod "4d6fdd92-d7a6-4b6e-85c5-04a94152335b" (UID: "4d6fdd92-d7a6-4b6e-85c5-04a94152335b"). InnerVolumeSpecName "kube-api-access-5g2bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:43.923197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:43.923178    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca" pod "4d6fdd92-d7a6-4b6e-85c5-04a94152335b" (UID: "4d6fdd92-d7a6-4b6e-85c5-04a94152335b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:44.022876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.022840    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g2bk\" (UniqueName: \"kubernetes.io/projected/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-kube-api-access-5g2bk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:44.022876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.022874    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6fdd92-d7a6-4b6e-85c5-04a94152335b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:44.600086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.600031    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" event={"ID":"0b64f18b-9634-486a-9ec2-a52afa35fd99","Type":"ContainerStarted","Data":"75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635"}
Oct 30 21:26:44.600275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.600113    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" event={"ID":"0b64f18b-9634-486a-9ec2-a52afa35fd99","Type":"ContainerStarted","Data":"1a111b34f2c17d5a1fc26b2945fce53f27d983418757d1406e650f3a76e25932"}
Oct 30 21:26:44.602005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.601978    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"
Oct 30 21:26:44.602131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.602008    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr" event={"ID":"4d6fdd92-d7a6-4b6e-85c5-04a94152335b","Type":"ContainerDied","Data":"97e9e693e21810c23784362d24c22c9c9503bbdad9a2e44e2d6d6f175adc780d"}
Oct 30 21:26:44.602131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.602054    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e9e693e21810c23784362d24c22c9c9503bbdad9a2e44e2d6d6f175adc780d"
Oct 30 21:26:44.684744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:44.684644    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podStartSLOduration=1.684596505 podStartE2EDuration="1.684596505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:44.67147783 +0000 UTC m=+987.593936153" watchObservedRunningTime="2026-10-30 21:26:44.684596505 +0000 UTC m=+987.607054804"
Oct 30 21:26:45.553228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:45.553190    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:45.554954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:45.554931    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:26:45.555058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:45.554991    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 21:26:46.558625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:46.556000    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:26:46.558625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:46.556064    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 21:26:47.553290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:47.553243    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:26:47.553480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:47.553325    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 21:26:48.553396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:48.553355    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:26:48.553831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:48.553422    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 21:26:49.552949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:49.552917    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:26:49.553109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:49.552978    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 21:26:50.552684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.552647    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:26:50.553189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.552708    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 21:26:50.794495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.794456    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"]
Oct 30 21:26:50.795097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.795083    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4d6fdd92-d7a6-4b6e-85c5-04a94152335b" containerName="revoke-keys"
Oct 30 21:26:50.795158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.795098    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6fdd92-d7a6-4b6e-85c5-04a94152335b" containerName="revoke-keys"
Oct 30 21:26:50.795208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.795202    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6fdd92-d7a6-4b6e-85c5-04a94152335b" containerName="revoke-keys"
Oct 30 21:26:50.800098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.800078    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.815459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.815425    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"]
Oct 30 21:26:50.821108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.821075    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:50.821339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.821095    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:50.821339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.821219    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:26:50.825904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.825868    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:50.883301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.883264    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.883460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.883399    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprmj\" (UniqueName: \"kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.883460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.883449    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.984828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.984760    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.985222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.985197    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.985328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.985305    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprmj\" (UniqueName: \"kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.989648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.988087    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:50.989648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:50.988908    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:51.008904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.003961    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprmj\" (UniqueName: \"kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj\") pod \"maas-api-revoke-sub-e8498fa7-vnwnz\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:51.117353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.117267    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:51.440000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.439970    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"]
Oct 30 21:26:51.441627 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:26:51.441587    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2ac0a9_4f99_42c4_b546_185fcacf1084.slice/crio-d4b02d8abc4d32bbca135e20e6c966d917a6407d17e4bfd54b3246e7642d55cb WatchSource:0}: Error finding container d4b02d8abc4d32bbca135e20e6c966d917a6407d17e4bfd54b3246e7642d55cb: Status 404 returned error can't find the container with id d4b02d8abc4d32bbca135e20e6c966d917a6407d17e4bfd54b3246e7642d55cb
Oct 30 21:26:51.558328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.558300    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:51.558836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.558817    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:51.559492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.559468    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:26:51.708653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:51.708532    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz" event={"ID":"1b2ac0a9-4f99-42c4-b546-185fcacf1084","Type":"ContainerStarted","Data":"d4b02d8abc4d32bbca135e20e6c966d917a6407d17e4bfd54b3246e7642d55cb"}
Oct 30 21:26:52.717689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:52.717558    6695 generic.go:356] "Generic (PLEG): container finished" podID="1b2ac0a9-4f99-42c4-b546-185fcacf1084" containerID="8c97863cea9534daa350d80971aacdad86813508dbb78add6c189392e384cbbe" exitCode=0
Oct 30 21:26:52.717689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:52.717645    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz" event={"ID":"1b2ac0a9-4f99-42c4-b546-185fcacf1084","Type":"ContainerDied","Data":"8c97863cea9534daa350d80971aacdad86813508dbb78add6c189392e384cbbe"}
Oct 30 21:26:53.899693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.899663    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:53.950561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.950521    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca\") pod \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") "
Oct 30 21:26:53.950812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.950791    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj\" (UniqueName: \"kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj\") pod \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") "
Oct 30 21:26:53.950877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.950835    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca\") pod \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\" (UID: \"1b2ac0a9-4f99-42c4-b546-185fcacf1084\") "
Oct 30 21:26:53.950970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.950942    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca" pod "1b2ac0a9-4f99-42c4-b546-185fcacf1084" (UID: "1b2ac0a9-4f99-42c4-b546-185fcacf1084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:53.951556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.951497    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:53.953118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.953093    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca" pod "1b2ac0a9-4f99-42c4-b546-185fcacf1084" (UID: "1b2ac0a9-4f99-42c4-b546-185fcacf1084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:53.953118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:53.953101    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj" pod "1b2ac0a9-4f99-42c4-b546-185fcacf1084" (UID: "1b2ac0a9-4f99-42c4-b546-185fcacf1084"). InnerVolumeSpecName "kube-api-access-dprmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:54.053189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:54.053147    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dprmj\" (UniqueName: \"kubernetes.io/projected/1b2ac0a9-4f99-42c4-b546-185fcacf1084-kube-api-access-dprmj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:54.053189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:54.053183    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2ac0a9-4f99-42c4-b546-185fcacf1084-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:26:54.735149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:54.735116    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"
Oct 30 21:26:54.735328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:54.735164    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz" event={"ID":"1b2ac0a9-4f99-42c4-b546-185fcacf1084","Type":"ContainerDied","Data":"d4b02d8abc4d32bbca135e20e6c966d917a6407d17e4bfd54b3246e7642d55cb"}
Oct 30 21:26:54.735328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:54.735223    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4b02d8abc4d32bbca135e20e6c966d917a6407d17e4bfd54b3246e7642d55cb"
Oct 30 21:26:57.407628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:57.402704    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:57.507628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:57.506768    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:57.545626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:57.541766    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:26:57.763685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:57.762706    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:57.856629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:57.855849    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:26:57.879632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:57.865540    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.482629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.434842    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.482629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.435629    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:26:58.581707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.536144    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.581707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.537266    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:26:58.787694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.601670    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:26:58.787694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.605066    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:26:58.787694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.765729    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.787694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.781962    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:26:58.877696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.872210    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.071734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.879069    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:26:59.071734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:58.956671    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.071734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.025832    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:26:59.111631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.101815    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:59.202628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.101911    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.553628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.551525    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.633743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.593684    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.697744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.601721    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmvr9" podUID="583d659a-c61c-4378-a025-f370c0dd1472" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.815683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.607253    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.816866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.771270    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.887627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.880798    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.960629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:26:59.959641    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.555630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:00.554701    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:27:00.706638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:00.703139    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.837686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:00.823706    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.837686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:00.824104    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:27:00.922693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:00.895697    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.974803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:00.974753    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:27:01.555533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.555454    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:27:01.605290 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:27:01.605248    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:27:01.660755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.660704    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.840933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.839248    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:27:01.840933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.839813    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.840933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.839765    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.936631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.927731    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.980902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:01.980846    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:02.562588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:02.562543    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:27:02.852685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:02.846905    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:02.852685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:02.847394    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:02.852685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:02.848878    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:27:02.932653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:02.931201    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:02.994657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:02.991247    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:27:03.203655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.203507    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:03.239845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.239814    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:27:03.240461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.240440    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:27:03.372095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.372052    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:03.577627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.577195    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:27:03.850890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.850767    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:27:03.851771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.851734    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:03.869103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.869062    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:27:03.940529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.936844    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:27:03.966027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.959859    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:27:03.983923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:03.983889    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:27:04.313624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.310958    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"]
Oct 30 21:27:04.313624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.312777    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1b2ac0a9-4f99-42c4-b546-185fcacf1084" containerName="revoke-keys"
Oct 30 21:27:04.313624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.312800    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2ac0a9-4f99-42c4-b546-185fcacf1084" containerName="revoke-keys"
Oct 30 21:27:04.313624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.313023    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2ac0a9-4f99-42c4-b546-185fcacf1084" containerName="revoke-keys"
Oct 30 21:27:04.396150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.396114    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"]
Oct 30 21:27:04.396533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.396513    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.413643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.410752    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-659efa-openshift-default-dockercfg-q7mzp\""
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511444    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511518    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511553    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511685    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511738    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511828    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511934    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.511966    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.520627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.512041    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmgg\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613442    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613636    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613695    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613790    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613941    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.613968    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.614025    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmgg\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.614080    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.619281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.615077    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.637546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.635347    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.637546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.635651    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.637546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.636305    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.637546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.637104    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.649412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.649379    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.670773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.670685    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.676673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.674513    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.683592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.683528    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmgg\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg\") pod \"e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") " pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:04.718205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:04.717907    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:05.096857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.096826    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"]
Oct 30 21:27:05.098999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:05.098969    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3ab0c9_4321_41d1_a2b2_771cdb5dd53b.slice/crio-fef02cb826079fc0d523ad2d929091c1478aedc3d3421798dbcf0ff4c9715800 WatchSource:0}: Error finding container fef02cb826079fc0d523ad2d929091c1478aedc3d3421798dbcf0ff4c9715800: Status 404 returned error can't find the container with id fef02cb826079fc0d523ad2d929091c1478aedc3d3421798dbcf0ff4c9715800
Oct 30 21:27:05.103674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.102152    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:05.103674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.102305    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:05.103674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.102398    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:05.676904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.676866    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" event={"ID":"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b","Type":"ContainerStarted","Data":"617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"}
Oct 30 21:27:05.677105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.676923    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" event={"ID":"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b","Type":"ContainerStarted","Data":"fef02cb826079fc0d523ad2d929091c1478aedc3d3421798dbcf0ff4c9715800"}
Oct 30 21:27:05.719816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.719780    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:05.731362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:05.731313    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podStartSLOduration=1.7312952780000002 podStartE2EDuration="1.731295278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:05.728835486 +0000 UTC m=+1008.651293784" watchObservedRunningTime="2026-10-30 21:27:05.731295278 +0000 UTC m=+1008.653753575"
Oct 30 21:27:06.738119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:06.737190    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:06.742659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:06.742626    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:06.743583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:06.743563    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:10.986934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:10.986897    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"]
Oct 30 21:27:10.991783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:10.991743    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.012027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.011995    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"]
Oct 30 21:27:11.107628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.104074    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-659efa-dockercfg-z62kr\""
Oct 30 21:27:11.107628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.104176    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-659efa\""
Oct 30 21:27:11.281019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.280977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume\") pod \"payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") " pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.281448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.281421    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht62m\" (UniqueName: \"kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m\") pod \"payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") " pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.382703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.382657    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht62m\" (UniqueName: \"kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m\") pod \"payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") " pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.382918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.382900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume\") pod \"payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") " pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.383719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.383696    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume\") pod \"payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") " pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.403002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.402975    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht62m\" (UniqueName: \"kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m\") pod \"payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") " pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.608366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.608278    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:11.925543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:11.925511    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"]
Oct 30 21:27:11.927717 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:11.927675    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd49aa86_1888_4064_982d_b26fbc610481.slice/crio-1a5a5cf72c517ed257412a1780d8744076f3c4e570347ed194c42cdee187e3c0 WatchSource:0}: Error finding container 1a5a5cf72c517ed257412a1780d8744076f3c4e570347ed194c42cdee187e3c0: Status 404 returned error can't find the container with id 1a5a5cf72c517ed257412a1780d8744076f3c4e570347ed194c42cdee187e3c0
Oct 30 21:27:12.844015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:12.843962    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm" event={"ID":"fd49aa86-1888-4064-982d-b26fbc610481","Type":"ContainerStarted","Data":"a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208"}
Oct 30 21:27:12.844626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:12.844579    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm" event={"ID":"fd49aa86-1888-4064-982d-b26fbc610481","Type":"ContainerStarted","Data":"1a5a5cf72c517ed257412a1780d8744076f3c4e570347ed194c42cdee187e3c0"}
Oct 30 21:27:12.846244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:12.846221    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:13.090627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:13.087437    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm" podStartSLOduration=3.087415217 podStartE2EDuration="3.087415217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:12.900512266 +0000 UTC m=+1015.822970565" watchObservedRunningTime="2026-10-30 21:27:13.087415217 +0000 UTC m=+1016.009873515"
Oct 30 21:27:17.498951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:17.498920    6695 scope.go:122] "RemoveContainer" containerID="c6b36fb36a0da434f4a62e84a4c337fcf979f022f4d6fdec43b9f0e8488700fe"
Oct 30 21:27:17.525552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:17.525520    6695 scope.go:122] "RemoveContainer" containerID="e96fa95ce8e656b5a7c3ea189f3f3bcf4582e8b8f5e396d5c101cb9ec37834c2"
Oct 30 21:27:17.561627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:17.558816    6695 scope.go:122] "RemoveContainer" containerID="39bd3687c79cc85809e0d12da3787c19db464e7425983525bf73caffaf35b9cb"
Oct 30 21:27:17.606692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:17.606094    6695 scope.go:122] "RemoveContainer" containerID="d857b119880051a3cc6701a7697842d8613640bae97d378b52e2e5d9102045cf"
Oct 30 21:27:17.698755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:17.698334    6695 scope.go:122] "RemoveContainer" containerID="ae73ff529239c24610ef4676b500cef6164813a325704d1682b4ef73134675b0"
Oct 30 21:27:24.898370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:24.898342    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:27:30.501926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.501890    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"]
Oct 30 21:27:30.508314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.508291    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.509814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.509787    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.509933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.509845    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82jv\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.509933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.509908    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.510036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.509934    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.510036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.509969    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.510154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.510124    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.510205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.510185    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.510261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.510232    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.510362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.510346    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.528699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.528657    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"]
Oct 30 21:27:30.611278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611244    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611297    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82jv\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611334    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611357    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611425    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611495    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611542    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611567    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611657    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611747    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611770    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611873    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.611993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.611925    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.612312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.612281    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.614008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.613981    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.614183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.614167    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.630469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.630430    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.630640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.630483    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82jv\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:30.825156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:30.825123    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:31.074788 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:27:31.074747    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:27:31.153923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:31.153882    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"]
Oct 30 21:27:31.155493 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:31.155467    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be073e4_66f5_4635_b73c_d9901f1c155e.slice/crio-9d6e0986cb8946c4ec617afb4222aa38f14a020087266d3912412f0296233474 WatchSource:0}: Error finding container 9d6e0986cb8946c4ec617afb4222aa38f14a020087266d3912412f0296233474: Status 404 returned error can't find the container with id 9d6e0986cb8946c4ec617afb4222aa38f14a020087266d3912412f0296233474
Oct 30 21:27:31.158132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:31.158099    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:31.158229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:31.158165    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:31.158229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:31.158194    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:32.092231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:32.092187    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" event={"ID":"1be073e4-66f5-4635-b73c-d9901f1c155e","Type":"ContainerStarted","Data":"09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864"}
Oct 30 21:27:32.092644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:32.092253    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" event={"ID":"1be073e4-66f5-4635-b73c-d9901f1c155e","Type":"ContainerStarted","Data":"9d6e0986cb8946c4ec617afb4222aa38f14a020087266d3912412f0296233474"}
Oct 30 21:27:32.161876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:32.161821    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podStartSLOduration=2.161805833 podStartE2EDuration="2.161805833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:32.15852484 +0000 UTC m=+1035.080983138" watchObservedRunningTime="2026-10-30 21:27:32.161805833 +0000 UTC m=+1035.084264198"
Oct 30 21:27:32.830624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:32.827960    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:32.836666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:32.835908    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:32.836666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:32.835984    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:33.833628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:33.832704    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:33.833628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:33.832766    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:34.829774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:34.829740    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:34.830071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:34.830038    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:35.829910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:35.829867    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:35.830494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:35.830460    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:36.833679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:36.827852    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:36.833679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:36.827912    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:37.827019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:37.826977    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:37.828047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:37.828015    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:38.826672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:38.826042    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:27:38.826672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:38.826122    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 21:27:40.868782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:40.868738    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:27:40.869267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:40.868825    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:41.120424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.114936    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"]
Oct 30 21:27:41.156977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.156940    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.213054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.213016    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-834924-openshift-default-dockercfg-vrw2t\""
Oct 30 21:27:41.271631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.220433    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.276750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.276713    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.276998    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62z72\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.277185    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.277282    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.277425    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.277527    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.277629    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.277748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.277737    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.437763    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.437997    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438044    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62z72\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438187    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438225    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438357    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438391    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.438427    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.452686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.449155    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.504626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.476031    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.504910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.502885    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.512915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.499269    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.592265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.557964    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.600427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.600393    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"]
Oct 30 21:27:41.630628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.629815    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62z72\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.665625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.664909    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.665820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.665803    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.717685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.716888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token\") pod \"e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") " pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.827965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.827919    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:41.939627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.936691    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:27:41.939627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.936771    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:41.948695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.947928    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:41.949730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.949465    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:41.967590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:41.967557    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:27:42.302716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.302689    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"]
Oct 30 21:27:42.304934 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:42.304906    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251d9852_8145_4cc5_a9ea_7e295dc460c0.slice/crio-16d67a524e904e6ab85f80028487590494f7e39d75785c4ed971cf72dd7264c8 WatchSource:0}: Error finding container 16d67a524e904e6ab85f80028487590494f7e39d75785c4ed971cf72dd7264c8: Status 404 returned error can't find the container with id 16d67a524e904e6ab85f80028487590494f7e39d75785c4ed971cf72dd7264c8
Oct 30 21:27:42.307587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.307553    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:42.307720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.307671    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:42.307786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.307724    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:42.654896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.654785    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" event={"ID":"251d9852-8145-4cc5-a9ea-7e295dc460c0","Type":"ContainerStarted","Data":"7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"}
Oct 30 21:27:42.654896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.654865    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" event={"ID":"251d9852-8145-4cc5-a9ea-7e295dc460c0","Type":"ContainerStarted","Data":"16d67a524e904e6ab85f80028487590494f7e39d75785c4ed971cf72dd7264c8"}
Oct 30 21:27:42.703776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.703723    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podStartSLOduration=1.7037067970000002 podStartE2EDuration="1.703706797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:42.696057207 +0000 UTC m=+1045.618515506" watchObservedRunningTime="2026-10-30 21:27:42.703706797 +0000 UTC m=+1045.626165095"
Oct 30 21:27:42.836643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:42.830698    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:43.849653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:43.849003    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:44.190134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.190101    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"]
Oct 30 21:27:44.194668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.194645    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.211321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.211297    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:27:44.211810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.211794    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:27:44.211863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.211837    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:27:44.211898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.211875    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:27:44.213243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.213214    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"]
Oct 30 21:27:44.271880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.271851    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.272052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.271977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.272190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.272173    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2lk\" (UniqueName: \"kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.373665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.373628    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2lk\" (UniqueName: \"kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.373839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.373728    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.373839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.373816    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.374506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.374475    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.376167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.376142    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.392919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.392852    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2lk\" (UniqueName: \"kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk\") pod \"maas-api-revoke-sub-f39cf64a-s8t4q\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.512049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.512012    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:44.794995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.794958    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:44.796145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.796116    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:27:44.819817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:44.819794    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"]
Oct 30 21:27:44.821941 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:44.821913    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4718f6_dc6f_4e9a_a6ea_2dcdcac68580.slice/crio-6ef6e28c3e136291784882c9cfe7c66c05d7251650f84ff773f90f8ffd00186d WatchSource:0}: Error finding container 6ef6e28c3e136291784882c9cfe7c66c05d7251650f84ff773f90f8ffd00186d: Status 404 returned error can't find the container with id 6ef6e28c3e136291784882c9cfe7c66c05d7251650f84ff773f90f8ffd00186d
Oct 30 21:27:45.805776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:45.805731    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q" event={"ID":"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580","Type":"ContainerStarted","Data":"64db2fbc031f5c207c7abc466b4209b45797a3c0f313c007bcb7350ec8ebf8f4"}
Oct 30 21:27:45.806320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:45.805804    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q" event={"ID":"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580","Type":"ContainerStarted","Data":"6ef6e28c3e136291784882c9cfe7c66c05d7251650f84ff773f90f8ffd00186d"}
Oct 30 21:27:45.847787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:45.847728    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q" podStartSLOduration=1.847711807 podStartE2EDuration="1.847711807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:45.843591793 +0000 UTC m=+1048.766050154" watchObservedRunningTime="2026-10-30 21:27:45.847711807 +0000 UTC m=+1048.770170104"
Oct 30 21:27:46.815305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.815272    6695 generic.go:356] "Generic (PLEG): container finished" podID="fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" containerID="64db2fbc031f5c207c7abc466b4209b45797a3c0f313c007bcb7350ec8ebf8f4" exitCode=0
Oct 30 21:27:46.817550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.817519    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q" event={"ID":"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580","Type":"ContainerDied","Data":"64db2fbc031f5c207c7abc466b4209b45797a3c0f313c007bcb7350ec8ebf8f4"}
Oct 30 21:27:46.960312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.960269    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"]
Oct 30 21:27:46.965788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.965765    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:46.982278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.982247    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"]
Oct 30 21:27:46.982663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.982632    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-834924\""
Oct 30 21:27:46.983854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:46.983824    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-834924-dockercfg-45pg7\""
Oct 30 21:27:47.174036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.143905    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.174036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.143953    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjvx\" (UniqueName: \"kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx\") pod \"payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.307267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.283864    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.308566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.308091    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjvx\" (UniqueName: \"kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx\") pod \"payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.309591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.309559    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.437249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.437121    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjvx\" (UniqueName: \"kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx\") pod \"payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.521921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.521876    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"]
Oct 30 21:27:47.536246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.536210    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"]
Oct 30 21:27:47.536404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.536338    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.588448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.588417    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:47.682260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.681787    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") " pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.682260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.682064    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl98k\" (UniqueName: \"kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k\") pod \"payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") " pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.783296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.783256    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") " pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.783515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.783494    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl98k\" (UniqueName: \"kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k\") pod \"payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") " pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.784088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.784068    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") " pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.802710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.802679    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl98k\" (UniqueName: \"kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k\") pod \"payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") " pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.857740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.857701    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:47.921901 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:47.921869    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode140e7b1_469a_4872_83a1_8327f9ebfa76.slice/crio-45a6248760b7b90cfd6de8e0a352678c68d05ebfa4bdc16903c3b1bd7b5ab2e1 WatchSource:0}: Error finding container 45a6248760b7b90cfd6de8e0a352678c68d05ebfa4bdc16903c3b1bd7b5ab2e1: Status 404 returned error can't find the container with id 45a6248760b7b90cfd6de8e0a352678c68d05ebfa4bdc16903c3b1bd7b5ab2e1
Oct 30 21:27:47.931463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:47.930661    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"]
Oct 30 21:27:48.034464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.034390    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:48.088675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.088645    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca\") pod \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") "
Oct 30 21:27:48.088675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.088689    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk\" (UniqueName: \"kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk\") pod \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") "
Oct 30 21:27:48.088950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.088818    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca\") pod \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\" (UID: \"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580\") "
Oct 30 21:27:48.089538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.089506    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca" pod "fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" (UID: "fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:48.092216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.092189    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca" pod "fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" (UID: "fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:27:48.092310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.092224    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk" pod "fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" (UID: "fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580"). InnerVolumeSpecName "kube-api-access-pm2lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:48.191575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.191513    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:27:48.191575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.191576    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2lk\" (UniqueName: \"kubernetes.io/projected/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-kube-api-access-pm2lk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:27:48.191937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.191592    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:27:48.195443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.195414    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"]
Oct 30 21:27:48.199975 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:48.199949    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f2e646_7a58_426a_b395_a7102953651d.slice/crio-e2c3a88ff835b53b54d8806211ac7e2a99df5e4548037fc3637d8330d9a45c99 WatchSource:0}: Error finding container e2c3a88ff835b53b54d8806211ac7e2a99df5e4548037fc3637d8330d9a45c99: Status 404 returned error can't find the container with id e2c3a88ff835b53b54d8806211ac7e2a99df5e4548037fc3637d8330d9a45c99
Oct 30 21:27:48.835491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.835455    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz" event={"ID":"e140e7b1-469a-4872-83a1-8327f9ebfa76","Type":"ContainerStarted","Data":"164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593"}
Oct 30 21:27:48.835751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.835517    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz" event={"ID":"e140e7b1-469a-4872-83a1-8327f9ebfa76","Type":"ContainerStarted","Data":"45a6248760b7b90cfd6de8e0a352678c68d05ebfa4bdc16903c3b1bd7b5ab2e1"}
Oct 30 21:27:48.835751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.835692    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:27:48.837457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.837429    6695 generic.go:356] "Generic (PLEG): container finished" podID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerID="617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f" exitCode=0
Oct 30 21:27:48.837633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.837505    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" event={"ID":"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b","Type":"ContainerDied","Data":"617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"}
Oct 30 21:27:48.838008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.837977    6695 scope.go:122] "RemoveContainer" containerID="617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"
Oct 30 21:27:48.839495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.839469    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"
Oct 30 21:27:48.839495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.839489    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q" event={"ID":"fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580","Type":"ContainerDied","Data":"6ef6e28c3e136291784882c9cfe7c66c05d7251650f84ff773f90f8ffd00186d"}
Oct 30 21:27:48.839656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.839525    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ef6e28c3e136291784882c9cfe7c66c05d7251650f84ff773f90f8ffd00186d"
Oct 30 21:27:48.839715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.839661    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:48.839770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.839728    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:48.839770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.839765    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:48.842673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.842646    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r" event={"ID":"86f2e646-7a58-426a-b395-a7102953651d","Type":"ContainerStarted","Data":"2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5"}
Oct 30 21:27:48.842791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.842697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r" event={"ID":"86f2e646-7a58-426a-b395-a7102953651d","Type":"ContainerStarted","Data":"e2c3a88ff835b53b54d8806211ac7e2a99df5e4548037fc3637d8330d9a45c99"}
Oct 30 21:27:48.842791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.842730    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:48.876197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.876131    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz" podStartSLOduration=2.876112682 podStartE2EDuration="2.876112682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:48.874413261 +0000 UTC m=+1051.796871559" watchObservedRunningTime="2026-10-30 21:27:48.876112682 +0000 UTC m=+1051.798570981"
Oct 30 21:27:48.971451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:48.971311    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r" podStartSLOduration=1.971292144 podStartE2EDuration="1.971292144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:48.969533244 +0000 UTC m=+1051.891991543" watchObservedRunningTime="2026-10-30 21:27:48.971292144 +0000 UTC m=+1051.893750443"
Oct 30 21:27:49.856922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:49.856883    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" event={"ID":"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b","Type":"ContainerStarted","Data":"4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698"}
Oct 30 21:27:50.729701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:50.729665    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:50.736448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:50.736417    6695 patch_prober.go:29] interesting pod/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.736639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:50.736488    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:27:51.722065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:51.722028    6695 patch_prober.go:29] interesting pod/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.722375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:51.722343    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:27:51.742695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:51.742662    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:52.723064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:52.723023    6695 patch_prober.go:29] interesting pod/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.723382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:52.723352    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:27:53.043260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:53.043213    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"]
Oct 30 21:27:53.047527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:53.047496    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aad469c-6jnsl"]
Oct 30 21:27:53.234626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:53.234062    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e838aae8-cad3-46aa-9aad-a7f8813d0879" path="/var/lib/kubelet/pods/e838aae8-cad3-46aa-9aad-a7f8813d0879/volumes"
Oct 30 21:27:53.729636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:53.721628    6695 patch_prober.go:29] interesting pod/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.729636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:53.721690    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:27:54.728199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:54.728158    6695 patch_prober.go:29] interesting pod/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.728835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:54.728794    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 21:27:55.027250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:55.027212    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"]
Oct 30 21:27:55.038400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:55.038367    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238dcb05-hc87t"]
Oct 30 21:27:55.215560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:55.215525    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac" path="/var/lib/kubelet/pods/17ae3e32-0b1f-4b79-8ad6-e00a32aff5ac/volumes"
Oct 30 21:27:55.724824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:55.724783    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:55.726094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:55.726074    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:27:56.029343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.029307    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"]
Oct 30 21:27:56.034016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.033990    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cd37886-sk77x"]
Oct 30 21:27:56.167394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.167357    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"]
Oct 30 21:27:56.168498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.168466    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" containerName="revoke-keys"
Oct 30 21:27:56.168498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.168496    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" containerName="revoke-keys"
Oct 30 21:27:56.168766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.168678    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" containerName="revoke-keys"
Oct 30 21:27:56.174241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.174213    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.190284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.190254    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b13793-openshift-default-dockercfg-dd8ct\""
Oct 30 21:27:56.194476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.194445    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"]
Oct 30 21:27:56.197949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.197922    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198120    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198230    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l78t\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198316    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198345    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198370    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198418    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198473    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.198629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.198533    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.328734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.307701    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.333800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.333761    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.333980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.333894    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l78t\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.334049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.333986    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.334049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.334015    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.334049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.334040    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.334207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.334091    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.334207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.334169    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.334303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.334238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.337625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.335105    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.359626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.338561    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.359626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.339159    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.359626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.339403    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.362104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.361550    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.405419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.405348    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l78t\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.418470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.418440    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.420659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.420630    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.421696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.421676    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token\") pod \"e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") " pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.506306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.505912    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:56.992094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.992063    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"]
Oct 30 21:27:56.994089 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:27:56.994057    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef73eff5_5da0_41af_86f6_744c054a8ef8.slice/crio-afa2fc785392e347adcce7fec896fb8859f035a7eeea504e8a2e19a5db8e5b07 WatchSource:0}: Error finding container afa2fc785392e347adcce7fec896fb8859f035a7eeea504e8a2e19a5db8e5b07: Status 404 returned error can't find the container with id afa2fc785392e347adcce7fec896fb8859f035a7eeea504e8a2e19a5db8e5b07
Oct 30 21:27:56.996808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.996778    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:56.997336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.996854    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:56.997336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:56.996903    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:57.216297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:57.216266    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c21f2c-8b1e-4af4-8b39-390237da2653" path="/var/lib/kubelet/pods/d8c21f2c-8b1e-4af4-8b39-390237da2653/volumes"
Oct 30 21:27:57.921152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:57.921114    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" event={"ID":"ef73eff5-5da0-41af-86f6-744c054a8ef8","Type":"ContainerStarted","Data":"1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"}
Oct 30 21:27:57.921358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:57.921180    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" event={"ID":"ef73eff5-5da0-41af-86f6-744c054a8ef8","Type":"ContainerStarted","Data":"afa2fc785392e347adcce7fec896fb8859f035a7eeea504e8a2e19a5db8e5b07"}
Oct 30 21:27:57.976178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:57.976125    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podStartSLOduration=1.976111268 podStartE2EDuration="1.976111268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:57.97245716 +0000 UTC m=+1060.894915458" watchObservedRunningTime="2026-10-30 21:27:57.976111268 +0000 UTC m=+1060.898569613"
Oct 30 21:27:58.513989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:58.513955    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:58.534628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:58.530654    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:58.934662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:58.934498    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:58.935569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:58.935540    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:27:59.863728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:59.862753    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:27:59.864208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:27:59.863886    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:28:01.073899 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:28:01.073857    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:28:02.040505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.040460    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"]
Oct 30 21:28:02.052110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.052060    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd4fd4e-sm6jq"]
Oct 30 21:28:02.278877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.278844    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"]
Oct 30 21:28:02.283910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.283886    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.301527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.301493    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"]
Oct 30 21:28:02.312514 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.312482    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b13793\""
Oct 30 21:28:02.312734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.312550    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b13793-dockercfg-slvfq\""
Oct 30 21:28:02.454627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.406655    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.454627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.406930    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwht\" (UniqueName: \"kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht\") pod \"payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.546706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.517158    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwht\" (UniqueName: \"kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht\") pod \"payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.551767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.551711    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.564943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.555937    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.571630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.560637    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"]
Oct 30 21:28:02.668348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.668299    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.671818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.671788    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"]
Oct 30 21:28:02.673430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.673353    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwht\" (UniqueName: \"kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht\") pod \"payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.772497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.772462    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.772687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.772516    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjjc\" (UniqueName: \"kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc\") pod \"payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.874076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.873979    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.874076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.874034    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjjc\" (UniqueName: \"kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc\") pod \"payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.874764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.874742    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.893799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.893773    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjjc\" (UniqueName: \"kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc\") pod \"payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") " pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:02.899839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.899810    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:02.999272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:02.999236    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:03.231169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:03.231086    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50211ceb-5c13-4dc7-ab8b-44a2eba16e04" path="/var/lib/kubelet/pods/50211ceb-5c13-4dc7-ab8b-44a2eba16e04/volumes"
Oct 30 21:28:03.250198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:03.250161    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"]
Oct 30 21:28:03.252310 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:03.252284    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfd29ad_1c1b_47a9_a6da_c81cf3cd71b8.slice/crio-d81ccaeae490ec64fe48620c8dd2c4c0b6e7d2a9de55cc29fc5b29745b3d76cd WatchSource:0}: Error finding container d81ccaeae490ec64fe48620c8dd2c4c0b6e7d2a9de55cc29fc5b29745b3d76cd: Status 404 returned error can't find the container with id d81ccaeae490ec64fe48620c8dd2c4c0b6e7d2a9de55cc29fc5b29745b3d76cd
Oct 30 21:28:03.327422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:03.327397    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"]
Oct 30 21:28:03.329440 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:03.329410    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51f0fa4_9e4f_44fb_a9a5_6ccb0dee4d67.slice/crio-c26887e061e7aad1552abed67d5ecc6167948b61226a4dc79abad6c9fef158e6 WatchSource:0}: Error finding container c26887e061e7aad1552abed67d5ecc6167948b61226a4dc79abad6c9fef158e6: Status 404 returned error can't find the container with id c26887e061e7aad1552abed67d5ecc6167948b61226a4dc79abad6c9fef158e6
Oct 30 21:28:04.004490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.004446    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47" event={"ID":"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8","Type":"ContainerStarted","Data":"517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13"}
Oct 30 21:28:04.004740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.004507    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:04.004740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.004529    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47" event={"ID":"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8","Type":"ContainerStarted","Data":"d81ccaeae490ec64fe48620c8dd2c4c0b6e7d2a9de55cc29fc5b29745b3d76cd"}
Oct 30 21:28:04.006383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.006355    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d" event={"ID":"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67","Type":"ContainerStarted","Data":"761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69"}
Oct 30 21:28:04.006482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.006404    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d" event={"ID":"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67","Type":"ContainerStarted","Data":"c26887e061e7aad1552abed67d5ecc6167948b61226a4dc79abad6c9fef158e6"}
Oct 30 21:28:04.006482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.006432    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:04.042186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.042131    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47" podStartSLOduration=2.042116868 podStartE2EDuration="2.042116868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:04.041372156 +0000 UTC m=+1066.963830451" watchObservedRunningTime="2026-10-30 21:28:04.042116868 +0000 UTC m=+1066.964575165"
Oct 30 21:28:04.080570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:04.080510    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d" podStartSLOduration=2.080493842 podStartE2EDuration="2.080493842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:04.078310698 +0000 UTC m=+1067.000768997" watchObservedRunningTime="2026-10-30 21:28:04.080493842 +0000 UTC m=+1067.002952139"
Oct 30 21:28:08.029839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:08.029802    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"]
Oct 30 21:28:08.040999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:08.040969    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79c2e8-5s5zw"]
Oct 30 21:28:09.216032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:09.215993    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48ca0a4-e6b1-4d32-8f7e-459be7368682" path="/var/lib/kubelet/pods/a48ca0a4-e6b1-4d32-8f7e-459be7368682/volumes"
Oct 30 21:28:10.060105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.060067    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"]
Oct 30 21:28:10.063338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.063319    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.079852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.079820    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"]
Oct 30 21:28:10.086694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.086661    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:28:10.086694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.086661    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:28:10.086914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.086768    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:28:10.089123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.089100    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:28:10.115386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.115345    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.115630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.115433    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5kc\" (UniqueName: \"kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.115630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.115534    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.216634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.216582    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5kc\" (UniqueName: \"kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.217153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.216681    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.217153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.216975    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.217438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.217417    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.219538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.219511    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.236531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.236497    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5kc\" (UniqueName: \"kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc\") pod \"maas-api-revoke-sub-a5038dc8-4tvwj\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.377832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.377753    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:10.688362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:10.688328    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"]
Oct 30 21:28:10.688827 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:10.688797    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf4386a_b018_455f_b823_d83c049d1ea5.slice/crio-d4c8e44e198aef50328121eb2f1efd0d8d226b026e5f77e8111ee83c7599e359 WatchSource:0}: Error finding container d4c8e44e198aef50328121eb2f1efd0d8d226b026e5f77e8111ee83c7599e359: Status 404 returned error can't find the container with id d4c8e44e198aef50328121eb2f1efd0d8d226b026e5f77e8111ee83c7599e359
Oct 30 21:28:11.028274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:11.028248    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"]
Oct 30 21:28:11.039101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:11.039073    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91d102c-k2jzt"]
Oct 30 21:28:11.061981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:11.061939    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj" event={"ID":"8bf4386a-b018-455f-b823-d83c049d1ea5","Type":"ContainerStarted","Data":"8e393ae32ea200f22a9047f3fa70eac9ed230cf0d8162399af024ff85829946f"}
Oct 30 21:28:11.062151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:11.062002    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj" event={"ID":"8bf4386a-b018-455f-b823-d83c049d1ea5","Type":"ContainerStarted","Data":"d4c8e44e198aef50328121eb2f1efd0d8d226b026e5f77e8111ee83c7599e359"}
Oct 30 21:28:11.109628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:11.105522    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj" podStartSLOduration=1.105498313 podStartE2EDuration="1.105498313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:11.100565571 +0000 UTC m=+1074.023023882" watchObservedRunningTime="2026-10-30 21:28:11.105498313 +0000 UTC m=+1074.027956611"
Oct 30 21:28:11.237843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:11.237807    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d808b5-0424-4d9e-bada-5634c4713e8f" path="/var/lib/kubelet/pods/e9d808b5-0424-4d9e-bada-5634c4713e8f/volumes"
Oct 30 21:28:12.026353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:12.026317    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"]
Oct 30 21:28:12.037676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:12.037641    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdc6af8-4sk74"]
Oct 30 21:28:12.071287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:12.071254    6695 generic.go:356] "Generic (PLEG): container finished" podID="8bf4386a-b018-455f-b823-d83c049d1ea5" containerID="8e393ae32ea200f22a9047f3fa70eac9ed230cf0d8162399af024ff85829946f" exitCode=0
Oct 30 21:28:12.071454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:12.071338    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj" event={"ID":"8bf4386a-b018-455f-b823-d83c049d1ea5","Type":"ContainerDied","Data":"8e393ae32ea200f22a9047f3fa70eac9ed230cf0d8162399af024ff85829946f"}
Oct 30 21:28:13.216145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.216117    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f22cd8a-962a-4439-bd6c-f9aaf4cb386e" path="/var/lib/kubelet/pods/0f22cd8a-962a-4439-bd6c-f9aaf4cb386e/volumes"
Oct 30 21:28:13.242208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.242181    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:13.264701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.264670    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca\") pod \"8bf4386a-b018-455f-b823-d83c049d1ea5\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") "
Oct 30 21:28:13.264863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.264711    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc\" (UniqueName: \"kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc\") pod \"8bf4386a-b018-455f-b823-d83c049d1ea5\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") "
Oct 30 21:28:13.264957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.264935    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca\") pod \"8bf4386a-b018-455f-b823-d83c049d1ea5\" (UID: \"8bf4386a-b018-455f-b823-d83c049d1ea5\") "
Oct 30 21:28:13.265267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.265237    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca" pod "8bf4386a-b018-455f-b823-d83c049d1ea5" (UID: "8bf4386a-b018-455f-b823-d83c049d1ea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:13.265649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.265630    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:13.266887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.266864    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc" pod "8bf4386a-b018-455f-b823-d83c049d1ea5" (UID: "8bf4386a-b018-455f-b823-d83c049d1ea5"). InnerVolumeSpecName "kube-api-access-8v5kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:13.266968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.266898    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca" pod "8bf4386a-b018-455f-b823-d83c049d1ea5" (UID: "8bf4386a-b018-455f-b823-d83c049d1ea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:13.367409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.367323    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4386a-b018-455f-b823-d83c049d1ea5-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:13.367409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:13.367354    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v5kc\" (UniqueName: \"kubernetes.io/projected/8bf4386a-b018-455f-b823-d83c049d1ea5-kube-api-access-8v5kc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:14.088259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:14.088224    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"
Oct 30 21:28:14.088259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:14.088245    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj" event={"ID":"8bf4386a-b018-455f-b823-d83c049d1ea5","Type":"ContainerDied","Data":"d4c8e44e198aef50328121eb2f1efd0d8d226b026e5f77e8111ee83c7599e359"}
Oct 30 21:28:14.088471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:14.088288    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c8e44e198aef50328121eb2f1efd0d8d226b026e5f77e8111ee83c7599e359"
Oct 30 21:28:15.017142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:15.017107    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:28:15.017590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:15.017185    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:28:21.869970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.869929    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"]
Oct 30 21:28:21.870648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.870633    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8bf4386a-b018-455f-b823-d83c049d1ea5" containerName="revoke-keys"
Oct 30 21:28:21.870706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.870649    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf4386a-b018-455f-b823-d83c049d1ea5" containerName="revoke-keys"
Oct 30 21:28:21.870755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.870745    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf4386a-b018-455f-b823-d83c049d1ea5" containerName="revoke-keys"
Oct 30 21:28:21.874751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.874728    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:21.889069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.889038    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:28:21.892646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.892619    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"]
Oct 30 21:28:21.899086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.899057    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:28:21.899460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.899307    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:28:21.899578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.899374    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:28:21.942628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.913658    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:21.943058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.943026    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfcv\" (UniqueName: \"kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:21.943357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:21.943338    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.042066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.042031    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"]
Oct 30 21:28:22.044989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.044943    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.045170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.045099    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfcv\" (UniqueName: \"kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.045267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.045249    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.045948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.045921    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.046640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.046594    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58832e5-xcdjf"]
Oct 30 21:28:22.047395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.047372    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.072660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.072589    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfcv\" (UniqueName: \"kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv\") pod \"maas-api-revoke-keys-e2e-api-a-659efa-bzjmz\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.195072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.194981    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:22.585918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:22.585885    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"]
Oct 30 21:28:22.588472 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:22.588426    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ed046f_0f0f_4a68_9302_bec741a96dd8.slice/crio-36dcdf44c8632c6fd8def3e5172d3571602226372607413e01838179503b5673 WatchSource:0}: Error finding container 36dcdf44c8632c6fd8def3e5172d3571602226372607413e01838179503b5673: Status 404 returned error can't find the container with id 36dcdf44c8632c6fd8def3e5172d3571602226372607413e01838179503b5673
Oct 30 21:28:23.194587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:23.194414    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz" event={"ID":"90ed046f-0f0f-4a68-9302-bec741a96dd8","Type":"ContainerStarted","Data":"0bfe1e4553c0f08c36f37b2f6eb9b6d28b92f2a151d6553e4cce8cc19ef3519e"}
Oct 30 21:28:23.194587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:23.194483    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz" event={"ID":"90ed046f-0f0f-4a68-9302-bec741a96dd8","Type":"ContainerStarted","Data":"36dcdf44c8632c6fd8def3e5172d3571602226372607413e01838179503b5673"}
Oct 30 21:28:23.245743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:23.245507    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz" podStartSLOduration=2.245485317 podStartE2EDuration="2.245485317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:23.233015418 +0000 UTC m=+1086.155473719" watchObservedRunningTime="2026-10-30 21:28:23.245485317 +0000 UTC m=+1086.167943617"
Oct 30 21:28:23.248205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:23.248178    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f70c919-d28e-4ba0-90b3-f79e829bc93a" path="/var/lib/kubelet/pods/7f70c919-d28e-4ba0-90b3-f79e829bc93a/volumes"
Oct 30 21:28:24.205552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:24.205513    6695 generic.go:356] "Generic (PLEG): container finished" podID="90ed046f-0f0f-4a68-9302-bec741a96dd8" containerID="0bfe1e4553c0f08c36f37b2f6eb9b6d28b92f2a151d6553e4cce8cc19ef3519e" exitCode=0
Oct 30 21:28:24.205968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:24.205598    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz" event={"ID":"90ed046f-0f0f-4a68-9302-bec741a96dd8","Type":"ContainerDied","Data":"0bfe1e4553c0f08c36f37b2f6eb9b6d28b92f2a151d6553e4cce8cc19ef3519e"}
Oct 30 21:28:25.576627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.567457    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:25.693729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.693686    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca\") pod \"90ed046f-0f0f-4a68-9302-bec741a96dd8\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") "
Oct 30 21:28:25.693884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.693874    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca\") pod \"90ed046f-0f0f-4a68-9302-bec741a96dd8\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") "
Oct 30 21:28:25.693978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.693960    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv\" (UniqueName: \"kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv\") pod \"90ed046f-0f0f-4a68-9302-bec741a96dd8\" (UID: \"90ed046f-0f0f-4a68-9302-bec741a96dd8\") "
Oct 30 21:28:25.694287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.694253    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca" pod "90ed046f-0f0f-4a68-9302-bec741a96dd8" (UID: "90ed046f-0f0f-4a68-9302-bec741a96dd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:25.694768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.694744    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:25.696130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.696107    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv" pod "90ed046f-0f0f-4a68-9302-bec741a96dd8" (UID: "90ed046f-0f0f-4a68-9302-bec741a96dd8"). InnerVolumeSpecName "kube-api-access-csfcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:25.696215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.696187    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca" pod "90ed046f-0f0f-4a68-9302-bec741a96dd8" (UID: "90ed046f-0f0f-4a68-9302-bec741a96dd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:25.796717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.796678    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ed046f-0f0f-4a68-9302-bec741a96dd8-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:25.796717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:25.796719    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfcv\" (UniqueName: \"kubernetes.io/projected/90ed046f-0f0f-4a68-9302-bec741a96dd8-kube-api-access-csfcv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:26.249000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:26.248924    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"
Oct 30 21:28:26.249000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:26.248939    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz" event={"ID":"90ed046f-0f0f-4a68-9302-bec741a96dd8","Type":"ContainerDied","Data":"36dcdf44c8632c6fd8def3e5172d3571602226372607413e01838179503b5673"}
Oct 30 21:28:26.249187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:26.249002    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36dcdf44c8632c6fd8def3e5172d3571602226372607413e01838179503b5673"
Oct 30 21:28:29.796223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:29.796183    6695 patch_prober.go:29] interesting pod/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:28:29.796805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:29.796253    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:28:30.292171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:30.292139    6695 generic.go:356] "Generic (PLEG): container finished" podID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerID="7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf" exitCode=0
Oct 30 21:28:30.292358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:30.292218    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" event={"ID":"251d9852-8145-4cc5-a9ea-7e295dc460c0","Type":"ContainerDied","Data":"7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"}
Oct 30 21:28:30.292753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:30.292725    6695 scope.go:122] "RemoveContainer" containerID="7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"
Oct 30 21:28:30.294313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:30.294286    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:28:30.294368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:30.294354    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:28:30.294403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:30.294389    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:28:31.074304 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:28:31.074264    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:28:31.302413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:31.302375    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" event={"ID":"251d9852-8145-4cc5-a9ea-7e295dc460c0","Type":"ContainerStarted","Data":"de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b"}
Oct 30 21:28:31.832627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:31.831985    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:28:31.832627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:31.832121    6695 patch_prober.go:29] interesting pod/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:28:31.832627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:31.832180    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:28:32.340865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:32.340825    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"]
Oct 30 21:28:32.836083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:32.836038    6695 patch_prober.go:29] interesting pod/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:28:32.836641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:32.836114    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:28:33.829003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:33.828967    6695 patch_prober.go:29] interesting pod/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:28:33.829654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:33.829585    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:28:34.830628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:34.829711    6695 patch_prober.go:29] interesting pod/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:28:34.830628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:34.829773    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:28:35.837313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:35.837057    6695 patch_prober.go:29] interesting pod/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:28:35.837313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:35.837129    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 21:28:36.834139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:36.834107    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:28:36.834433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:36.834405    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:28:36.835151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:36.835132    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:28:37.347631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.345672    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm" podUID="fd49aa86-1888-4064-982d-b26fbc610481" containerName="payload-processing" containerID="cri-o://a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208" gracePeriod=25
Oct 30 21:28:37.756048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.756015    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:28:37.764366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.764338    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m\" (UniqueName: \"kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m\") pod \"fd49aa86-1888-4064-982d-b26fbc610481\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") "
Oct 30 21:28:37.764549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.764394    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume\") pod \"fd49aa86-1888-4064-982d-b26fbc610481\" (UID: \"fd49aa86-1888-4064-982d-b26fbc610481\") "
Oct 30 21:28:37.764839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.764808    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume" pod "fd49aa86-1888-4064-982d-b26fbc610481" (UID: "fd49aa86-1888-4064-982d-b26fbc610481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:37.764965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.764948    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd49aa86-1888-4064-982d-b26fbc610481-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:37.766410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.766390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m" pod "fd49aa86-1888-4064-982d-b26fbc610481" (UID: "fd49aa86-1888-4064-982d-b26fbc610481"). InnerVolumeSpecName "kube-api-access-ht62m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:37.866256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:37.866220    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht62m\" (UniqueName: \"kubernetes.io/projected/fd49aa86-1888-4064-982d-b26fbc610481-kube-api-access-ht62m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:38.394758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.394721    6695 generic.go:356] "Generic (PLEG): container finished" podID="fd49aa86-1888-4064-982d-b26fbc610481" containerID="a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208" exitCode=0
Oct 30 21:28:38.395238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.394779    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"
Oct 30 21:28:38.395238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.394805    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm" event={"ID":"fd49aa86-1888-4064-982d-b26fbc610481","Type":"ContainerDied","Data":"a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208"}
Oct 30 21:28:38.395238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.394870    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm" event={"ID":"fd49aa86-1888-4064-982d-b26fbc610481","Type":"ContainerDied","Data":"1a5a5cf72c517ed257412a1780d8744076f3c4e570347ed194c42cdee187e3c0"}
Oct 30 21:28:38.395238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.394900    6695 scope.go:122] "RemoveContainer" containerID="a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208"
Oct 30 21:28:38.406242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.406217    6695 scope.go:122] "RemoveContainer" containerID="a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208"
Oct 30 21:28:38.406531 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:28:38.406509    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208\": container with ID starting with a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208 not found: ID does not exist" containerID="a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208"
Oct 30 21:28:38.406573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.406539    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208"} err="failed to get container status \"a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208\": rpc error: code = NotFound desc = could not find container \"a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208\": container with ID starting with a431367571ac40b21b960312c40fa3975fa799d1c75e613bddf72073682a9208 not found: ID does not exist"
Oct 30 21:28:38.446237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.446199    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"]
Oct 30 21:28:38.457566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:38.457535    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-659efa-5868fb47c9-d7lfm"]
Oct 30 21:28:39.230570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:39.230523    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd49aa86-1888-4064-982d-b26fbc610481" path="/var/lib/kubelet/pods/fd49aa86-1888-4064-982d-b26fbc610481/volumes"
Oct 30 21:28:42.272626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.272255    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"]
Oct 30 21:28:42.274151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.273315    6695 cpu_manager.go:414] "containerMap: removing container" podUID="90ed046f-0f0f-4a68-9302-bec741a96dd8" containerName="revoke-keys"
Oct 30 21:28:42.274151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.273341    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ed046f-0f0f-4a68-9302-bec741a96dd8" containerName="revoke-keys"
Oct 30 21:28:42.274151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.273414    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fd49aa86-1888-4064-982d-b26fbc610481" containerName="payload-processing"
Oct 30 21:28:42.274151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.273422    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd49aa86-1888-4064-982d-b26fbc610481" containerName="payload-processing"
Oct 30 21:28:42.274151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.273618    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd49aa86-1888-4064-982d-b26fbc610481" containerName="payload-processing"
Oct 30 21:28:42.274151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.273644    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ed046f-0f0f-4a68-9302-bec741a96dd8" containerName="revoke-keys"
Oct 30 21:28:42.280325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.280298    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"]
Oct 30 21:28:42.280539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.280445    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.299486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.299408    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c6dab8\""
Oct 30 21:28:42.436666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.338836    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c6dab8-dockercfg-6trhw\""
Oct 30 21:28:42.552409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.551923    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.552917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.552822    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwqtl\" (UniqueName: \"kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl\") pod \"payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.623626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.607298    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"]
Oct 30 21:28:42.645670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.635065    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:42.694795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.675659    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwqtl\" (UniqueName: \"kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl\") pod \"payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.694795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.675729    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppbq\" (UniqueName: \"kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq\") pod \"payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:42.694795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.675860    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:42.694795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.675930    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.758968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.758554    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.808999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.808907    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwqtl\" (UniqueName: \"kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl\") pod \"payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.862486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.862440    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"]
Oct 30 21:28:42.869517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.868101    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppbq\" (UniqueName: \"kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq\") pod \"payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:42.869517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.868328    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:42.869517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.869431    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:42.944927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.940843    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:42.967007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:42.963437    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppbq\" (UniqueName: \"kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq\") pod \"payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") " pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:43.188419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:43.168965    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:43.851692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:43.827729    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"]
Oct 30 21:28:44.577698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:44.576552    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:44.826627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:44.819145    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:28:44.901641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:44.876975    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:44.901641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:44.877371    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:28:44.970270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:44.970227    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:45.103688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.092424    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"]
Oct 30 21:28:45.129794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.129753    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:45.205622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.193295    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk" event={"ID":"3173f1d6-9c04-4826-b329-004c73fb2958","Type":"ContainerStarted","Data":"9cd6939694014564e24370ae1b4fd1676ea7668ed9cbbd8f786639fbc4497ad0"}
Oct 30 21:28:45.590358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.588416    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:28:45.590358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.589045    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:28:45.611678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.610954    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 30 21:28:45.837302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.837262    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:28:45.838110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.838082    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:28:45.880950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.880843    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:45.889627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.882365    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:28:45.909556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.885931    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:28:45.910185 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.910162    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:28:45.934630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:45.927133    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:46.318670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.315372    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:28:46.538308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.538273    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:28:46.662687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.645949    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:28:46.662687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.656049    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk" event={"ID":"3173f1d6-9c04-4826-b329-004c73fb2958","Type":"ContainerStarted","Data":"d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065"}
Oct 30 21:28:46.662687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.656985    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:46.720667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.715161    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:28:46.766172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.766102    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm" event={"ID":"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da","Type":"ContainerStarted","Data":"81851cea810baa2ed88c16df462d233ecd42e296dceaa6d4c53a73008b42ffab"}
Oct 30 21:28:46.766563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.766506    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:46.807417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.807385    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:28:46.850704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.850589    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk" podStartSLOduration=4.850565699 podStartE2EDuration="4.850565699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:46.846846394 +0000 UTC m=+1109.769304693" watchObservedRunningTime="2026-10-30 21:28:46.850565699 +0000 UTC m=+1109.773024001"
Oct 30 21:28:46.996761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:46.996663    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm" podStartSLOduration=4.996648845 podStartE2EDuration="4.996648845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:46.992779805 +0000 UTC m=+1109.915238103" watchObservedRunningTime="2026-10-30 21:28:46.996648845 +0000 UTC m=+1109.919107144"
Oct 30 21:28:47.784677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:47.784634    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm" event={"ID":"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da","Type":"ContainerStarted","Data":"4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988"}
Oct 30 21:28:48.282469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.282438    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7400-9d1806"]
Oct 30 21:28:48.287489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.287465    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:48.302150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.302122    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7400-9d1806"]
Oct 30 21:28:48.308659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.308631    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vfm5w\""
Oct 30 21:28:48.340627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.337362    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhj6v\" (UniqueName: \"kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v\") pod \"test-curl-7400-9d1806\" (UID: \"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d\") " pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:48.441031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.440993    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhj6v\" (UniqueName: \"kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v\") pod \"test-curl-7400-9d1806\" (UID: \"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d\") " pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:48.459497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.459467    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhj6v\" (UniqueName: \"kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v\") pod \"test-curl-7400-9d1806\" (UID: \"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d\") " pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:48.603441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.603362    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:48.943817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.943787    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7400-9d1806"]
Oct 30 21:28:48.947100 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:48.947067    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a93d88a_ea7d_47a0_947c_c0ec93e0ce7d.slice/crio-c61e35f1c0a42f4720a38aa349551350cac8b447544b6124f29850490f6fdebb WatchSource:0}: Error finding container c61e35f1c0a42f4720a38aa349551350cac8b447544b6124f29850490f6fdebb: Status 404 returned error can't find the container with id c61e35f1c0a42f4720a38aa349551350cac8b447544b6124f29850490f6fdebb
Oct 30 21:28:48.949751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:48.949728    6695 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:28:49.806379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:49.806336    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7400-9d1806" event={"ID":"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d","Type":"ContainerStarted","Data":"c61e35f1c0a42f4720a38aa349551350cac8b447544b6124f29850490f6fdebb"}
Oct 30 21:28:51.583006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.582955    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"]
Oct 30 21:28:51.619007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.618560    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"]
Oct 30 21:28:51.619007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.618725    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.645030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.644998    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:28:51.645193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.645025    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:28:51.645193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.645121    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:28:51.645314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.645119    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:28:51.721342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.721300    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgl4\" (UniqueName: \"kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.721521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.721467    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.721700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.721686    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.822922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.822890    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.823118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.822943    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgl4\" (UniqueName: \"kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.823118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.823016    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.823758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.823731    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.825661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.825640    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.831263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.831223    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7400-9d1806" event={"ID":"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d","Type":"ContainerStarted","Data":"029cc0f363822e59649a5201c24fd0f3481b9193203d175f4523db85992a1e37"}
Oct 30 21:28:51.841414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.841387    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgl4\" (UniqueName: \"kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4\") pod \"maas-api-revoke-sub-642e3c3e-ww445\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:51.866336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.866283    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7400-9d1806" podStartSLOduration=1.476614669 podStartE2EDuration="3.866269387s" totalImagesPullingTime="2.389654718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:51.865438936 +0000 UTC m=+1114.787897235" watchObservedRunningTime="2026-10-30 21:28:51.866269387 +0000 UTC m=+1114.788727686"
Oct 30 21:28:51.964461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:51.964373    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:52.335828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:52.335802    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"]
Oct 30 21:28:52.337296 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:52.337265    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6f073e_dec0_4abb_a4ae_eaf1dc7fafd2.slice/crio-9749d9ed4cafaf3ffa07ca48546e2f4f766e08b506d717a26d4731a9d8fb76f4 WatchSource:0}: Error finding container 9749d9ed4cafaf3ffa07ca48546e2f4f766e08b506d717a26d4731a9d8fb76f4: Status 404 returned error can't find the container with id 9749d9ed4cafaf3ffa07ca48546e2f4f766e08b506d717a26d4731a9d8fb76f4
Oct 30 21:28:52.841629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:52.841575    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445" event={"ID":"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2","Type":"ContainerStarted","Data":"e7773d1e6e98c0dd5c868c289410680075a20eaf48b8423a05be20daba15f650"}
Oct 30 21:28:52.842094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:52.841648    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445" event={"ID":"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2","Type":"ContainerStarted","Data":"9749d9ed4cafaf3ffa07ca48546e2f4f766e08b506d717a26d4731a9d8fb76f4"}
Oct 30 21:28:52.882635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:52.882052    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7400-9d1806"]
Oct 30 21:28:52.893644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:52.890666    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7400-9d1806"]
Oct 30 21:28:52.893644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:52.890962    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445" podStartSLOduration=1.890946053 podStartE2EDuration="1.890946053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:52.881895281 +0000 UTC m=+1115.804353574" watchObservedRunningTime="2026-10-30 21:28:52.890946053 +0000 UTC m=+1115.813404346"
Oct 30 21:28:53.496452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.496420    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"]
Oct 30 21:28:53.497185 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.497168    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" containerName="test-curl-7400-9d1806"
Oct 30 21:28:53.497231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.497187    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" containerName="test-curl-7400-9d1806"
Oct 30 21:28:53.497321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.497312    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" containerName="test-curl-7400-9d1806"
Oct 30 21:28:53.500985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.500966    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.533853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.533580    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"]
Oct 30 21:28:53.669330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.669242    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.669490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.669363    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jln\" (UniqueName: \"kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.669490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.669457    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.771441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.771392    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jln\" (UniqueName: \"kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.771704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.771488    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.771704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.771699    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.772254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.772227    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.774074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.774050    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.793817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.793787    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jln\" (UniqueName: \"kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:53.857639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.857585    6695 generic.go:356] "Generic (PLEG): container finished" podID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerID="1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784" exitCode=0
Oct 30 21:28:53.858153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.857641    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" event={"ID":"ef73eff5-5da0-41af-86f6-744c054a8ef8","Type":"ContainerDied","Data":"1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"}
Oct 30 21:28:53.858322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.858285    6695 scope.go:122] "RemoveContainer" containerID="1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"
Oct 30 21:28:53.859915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.859880    6695 generic.go:356] "Generic (PLEG): container finished" podID="3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" containerID="e7773d1e6e98c0dd5c868c289410680075a20eaf48b8423a05be20daba15f650" exitCode=0
Oct 30 21:28:53.860042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.859961    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445" event={"ID":"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2","Type":"ContainerDied","Data":"e7773d1e6e98c0dd5c868c289410680075a20eaf48b8423a05be20daba15f650"}
Oct 30 21:28:53.860164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.860147    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7400-9d1806" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" containerName="test-curl-7400-9d1806" containerID="cri-o://029cc0f363822e59649a5201c24fd0f3481b9193203d175f4523db85992a1e37" gracePeriod=2
Oct 30 21:28:53.860247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.860220    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:28:53.860302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.860291    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:28:53.860357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.860325    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:28:53.869973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:53.869953    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:54.200996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:54.200959    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"]
Oct 30 21:28:54.202968 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:28:54.202933    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd07fc9_3fe5_452b_9832_e5065e597e8b.slice/crio-10bcf478434abbafc0cdd0381c9c25a9319a11076cf85f29c2951a76de8070bd WatchSource:0}: Error finding container 10bcf478434abbafc0cdd0381c9c25a9319a11076cf85f29c2951a76de8070bd: Status 404 returned error can't find the container with id 10bcf478434abbafc0cdd0381c9c25a9319a11076cf85f29c2951a76de8070bd
Oct 30 21:28:54.870681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:54.870638    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t" event={"ID":"9dd07fc9-3fe5-452b-9832-e5065e597e8b","Type":"ContainerStarted","Data":"246370eb3de0b93e51ac9fae87d9220fbdd838edf0ba624c8f3acffd6e45e63a"}
Oct 30 21:28:54.871151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:54.870702    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t" event={"ID":"9dd07fc9-3fe5-452b-9832-e5065e597e8b","Type":"ContainerStarted","Data":"10bcf478434abbafc0cdd0381c9c25a9319a11076cf85f29c2951a76de8070bd"}
Oct 30 21:28:54.873253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:54.873216    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" event={"ID":"ef73eff5-5da0-41af-86f6-744c054a8ef8","Type":"ContainerStarted","Data":"ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940"}
Oct 30 21:28:54.909798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:54.909736    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t" podStartSLOduration=1.9097084770000001 podStartE2EDuration="1.909708477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:54.907159609 +0000 UTC m=+1117.829617911" watchObservedRunningTime="2026-10-30 21:28:54.909708477 +0000 UTC m=+1117.832166774"
Oct 30 21:28:55.089474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.089447    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:55.207632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.207520    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4\" (UniqueName: \"kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4\") pod \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") "
Oct 30 21:28:55.207812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.207596    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca\") pod \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") "
Oct 30 21:28:55.207888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.207848    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca\") pod \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\" (UID: \"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2\") "
Oct 30 21:28:55.207985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.207959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca" pod "3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" (UID: "3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:55.208400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.208368    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:55.210075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.210051    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca" pod "3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" (UID: "3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:55.210198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.210075    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4" pod "3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" (UID: "3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2"). InnerVolumeSpecName "kube-api-access-mqgl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:55.310358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.310325    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:55.310358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.310358    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqgl4\" (UniqueName: \"kubernetes.io/projected/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2-kube-api-access-mqgl4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:55.507555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.507458    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:28:55.508666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.508637    6695 patch_prober.go:29] interesting pod/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:28:55.508790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.508699    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:28:55.884856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.884820    6695 generic.go:356] "Generic (PLEG): container finished" podID="9dd07fc9-3fe5-452b-9832-e5065e597e8b" containerID="246370eb3de0b93e51ac9fae87d9220fbdd838edf0ba624c8f3acffd6e45e63a" exitCode=0
Oct 30 21:28:55.885153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.885038    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t" event={"ID":"9dd07fc9-3fe5-452b-9832-e5065e597e8b","Type":"ContainerDied","Data":"246370eb3de0b93e51ac9fae87d9220fbdd838edf0ba624c8f3acffd6e45e63a"}
Oct 30 21:28:55.886849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.886828    6695 generic.go:356] "Generic (PLEG): container finished" podID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" containerID="029cc0f363822e59649a5201c24fd0f3481b9193203d175f4523db85992a1e37" exitCode=137
Oct 30 21:28:55.888379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.888361    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"
Oct 30 21:28:55.888462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.888395    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445" event={"ID":"3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2","Type":"ContainerDied","Data":"9749d9ed4cafaf3ffa07ca48546e2f4f766e08b506d717a26d4731a9d8fb76f4"}
Oct 30 21:28:55.888462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:55.888447    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9749d9ed4cafaf3ffa07ca48546e2f4f766e08b506d717a26d4731a9d8fb76f4"
Oct 30 21:28:56.572654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.562084    6695 patch_prober.go:29] interesting pod/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:28:56.572654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.562155    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:28:56.704692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.704664    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:56.851928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.851892    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v\" (UniqueName: \"kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v\") pod \"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d\" (UID: \"6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d\") "
Oct 30 21:28:56.855521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.855486    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v" pod "6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" (UID: "6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d"). InnerVolumeSpecName "kube-api-access-lhj6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:56.860542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.860504    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"]
Oct 30 21:28:56.869207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.869171    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659efa-bzjmz"]
Oct 30 21:28:56.898973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.898944    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:56.899417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.899400    6695 scope.go:122] "RemoveContainer" containerID="029cc0f363822e59649a5201c24fd0f3481b9193203d175f4523db85992a1e37"
Oct 30 21:28:56.946720 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:28:56.946668    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7400-9d1806\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:56.953447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:56.953415    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhj6v\" (UniqueName: \"kubernetes.io/projected/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d-kube-api-access-lhj6v\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:57.260641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.241739    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:57.261012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.251927    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" path="/var/lib/kubelet/pods/6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d/volumes"
Oct 30 21:28:57.275648 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:28:57.266082    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7400-9d1806\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="6a93d88a-ea7d-47a0-947c-c0ec93e0ce7d" pod="opendatahub/test-curl-7400-9d1806"
Oct 30 21:28:57.277260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.277226    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ed046f-0f0f-4a68-9302-bec741a96dd8" path="/var/lib/kubelet/pods/90ed046f-0f0f-4a68-9302-bec741a96dd8/volumes"
Oct 30 21:28:57.318298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.318259    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca\") pod \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") "
Oct 30 21:28:57.321852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.321817    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln\" (UniqueName: \"kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln\") pod \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") "
Oct 30 21:28:57.329636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.322067    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca\") pod \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\" (UID: \"9dd07fc9-3fe5-452b-9832-e5065e597e8b\") "
Oct 30 21:28:57.338478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.338442    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca" pod "9dd07fc9-3fe5-452b-9832-e5065e597e8b" (UID: "9dd07fc9-3fe5-452b-9832-e5065e597e8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:57.345669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.345639    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca" pod "9dd07fc9-3fe5-452b-9832-e5065e597e8b" (UID: "9dd07fc9-3fe5-452b-9832-e5065e597e8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:57.348869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.348837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln" pod "9dd07fc9-3fe5-452b-9832-e5065e597e8b" (UID: "9dd07fc9-3fe5-452b-9832-e5065e597e8b"). InnerVolumeSpecName "kube-api-access-x4jln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:57.436484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.436431    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:57.436484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.436478    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jln\" (UniqueName: \"kubernetes.io/projected/9dd07fc9-3fe5-452b-9832-e5065e597e8b-kube-api-access-x4jln\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:57.436484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.436494    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd07fc9-3fe5-452b-9832-e5065e597e8b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:28:57.507190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.507160    6695 patch_prober.go:29] interesting pod/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:28:57.507343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.507210    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:28:57.934627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.924943    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t" event={"ID":"9dd07fc9-3fe5-452b-9832-e5065e597e8b","Type":"ContainerDied","Data":"10bcf478434abbafc0cdd0381c9c25a9319a11076cf85f29c2951a76de8070bd"}
Oct 30 21:28:57.934627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.925010    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10bcf478434abbafc0cdd0381c9c25a9319a11076cf85f29c2951a76de8070bd"
Oct 30 21:28:57.934627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:57.925088    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"
Oct 30 21:28:58.507303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:58.507262    6695 patch_prober.go:29] interesting pod/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:28:58.507483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:58.507334    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:28:58.795864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:58.795838    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:28:58.796064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:58.795903    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:28:58.936658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:58.936199    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:28:59.507286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:59.507247    6695 patch_prober.go:29] interesting pod/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:28:59.507468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:28:59.507321    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:29:00.583636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:00.583578    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:29:00.586249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:00.586223    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:29:01.074222 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:01.074187    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:29:02.684467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.684424    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"]
Oct 30 21:29:02.685744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.685555    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9dd07fc9-3fe5-452b-9832-e5065e597e8b" containerName="revoke-keys"
Oct 30 21:29:02.685744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.685583    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd07fc9-3fe5-452b-9832-e5065e597e8b" containerName="revoke-keys"
Oct 30 21:29:02.685744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.685672    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" containerName="revoke-keys"
Oct 30 21:29:02.685744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.685681    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" containerName="revoke-keys"
Oct 30 21:29:02.686070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.685860    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" containerName="revoke-keys"
Oct 30 21:29:02.686070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.685880    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd07fc9-3fe5-452b-9832-e5065e597e8b" containerName="revoke-keys"
Oct 30 21:29:02.693931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.693903    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.711980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.711937    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"]
Oct 30 21:29:02.717645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.716085    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ab5774-openshift-default-dockercfg-4qj6s\""
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861264    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861382    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgr9\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861486    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861704    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861800    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861848    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.861875    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.862071    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.862631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.862321    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963646    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963699    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrgr9\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963738    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963865    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963918    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963949    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.963976    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.964038    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.964683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.964256    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.965709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.965677    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.970629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.966220    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.970629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.966472    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.970629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.969459    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:02.987626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:02.987325    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:03.019645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.007999    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:03.026304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.025381    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:03.050626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.048732    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:03.069624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.063598    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrgr9\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9\") pod \"e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") " pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:03.316324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.316286    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:03.644205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.644165    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"]
Oct 30 21:29:03.645192 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:29:03.645163    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f502ae_377f_48ef_b68f_21d116b0d87c.slice/crio-3876a0d82e0e3a09cadaeda6e38e9e5c14a619c978d2b6344d43e69b0bdff152 WatchSource:0}: Error finding container 3876a0d82e0e3a09cadaeda6e38e9e5c14a619c978d2b6344d43e69b0bdff152: Status 404 returned error can't find the container with id 3876a0d82e0e3a09cadaeda6e38e9e5c14a619c978d2b6344d43e69b0bdff152
Oct 30 21:29:03.647967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.647931    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:03.648057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.647997    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:03.648057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:03.648024    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:04.075547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.075504    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" event={"ID":"f3f502ae-377f-48ef-b68f-21d116b0d87c","Type":"ContainerStarted","Data":"586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"}
Oct 30 21:29:04.076057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.075622    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" event={"ID":"f3f502ae-377f-48ef-b68f-21d116b0d87c","Type":"ContainerStarted","Data":"3876a0d82e0e3a09cadaeda6e38e9e5c14a619c978d2b6344d43e69b0bdff152"}
Oct 30 21:29:04.084393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.084359    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"]
Oct 30 21:29:04.084640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.084612    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy" containerID="cri-o://4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698" gracePeriod=30
Oct 30 21:29:04.121710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.119751    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podStartSLOduration=2.119733816 podStartE2EDuration="2.119733816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:04.115865912 +0000 UTC m=+1127.038324211" watchObservedRunningTime="2026-10-30 21:29:04.119733816 +0000 UTC m=+1127.042192114"
Oct 30 21:29:04.317121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.317091    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:04.323162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.323128    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:04.842624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.841709    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"]
Oct 30 21:29:04.886884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.873886    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.917045    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.917135    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.917150    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.917202    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.939970    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.940294    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpgn\" (UniqueName: \"kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:04.941422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.940328    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:04.946496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:04.946442    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"]
Oct 30 21:29:05.042131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.042081    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpgn\" (UniqueName: \"kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.042311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.042138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.042311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.042226    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.043160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.043131    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.044696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.044674    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.061289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.061263    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpgn\" (UniqueName: \"kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn\") pod \"maas-api-revoke-keys-e2e-api-b-834924-w7w97\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.083650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.083592    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:05.084559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.084540    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:05.275432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.275402    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:05.582176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:05.582141    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"]
Oct 30 21:29:05.583715 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:29:05.583688    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bf0bd5_3d72_48ef_89f0_45f2efea3955.slice/crio-8179207a8c7e70c75181ff1a21fccbc20925d6111e28df8a405d25d3761743ee WatchSource:0}: Error finding container 8179207a8c7e70c75181ff1a21fccbc20925d6111e28df8a405d25d3761743ee: Status 404 returned error can't find the container with id 8179207a8c7e70c75181ff1a21fccbc20925d6111e28df8a405d25d3761743ee
Oct 30 21:29:06.094155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:06.094083    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97" event={"ID":"47bf0bd5-3d72-48ef-89f0-45f2efea3955","Type":"ContainerStarted","Data":"5455fd04cedf5d7de5abb71dac8d45441efe73d9fdad1fb29aac9169ed169996"}
Oct 30 21:29:06.094639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:06.094244    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97" event={"ID":"47bf0bd5-3d72-48ef-89f0-45f2efea3955","Type":"ContainerStarted","Data":"8179207a8c7e70c75181ff1a21fccbc20925d6111e28df8a405d25d3761743ee"}
Oct 30 21:29:06.131999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:06.131946    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97" podStartSLOduration=2.131933166 podStartE2EDuration="2.131933166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:06.131506038 +0000 UTC m=+1129.053964349" watchObservedRunningTime="2026-10-30 21:29:06.131933166 +0000 UTC m=+1129.054391463"
Oct 30 21:29:07.103530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:07.103497    6695 generic.go:356] "Generic (PLEG): container finished" podID="47bf0bd5-3d72-48ef-89f0-45f2efea3955" containerID="5455fd04cedf5d7de5abb71dac8d45441efe73d9fdad1fb29aac9169ed169996" exitCode=0
Oct 30 21:29:07.103962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:07.103591    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97" event={"ID":"47bf0bd5-3d72-48ef-89f0-45f2efea3955","Type":"ContainerDied","Data":"5455fd04cedf5d7de5abb71dac8d45441efe73d9fdad1fb29aac9169ed169996"}
Oct 30 21:29:08.335961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.335929    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:08.410240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.410141    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn\" (UniqueName: \"kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn\") pod \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") "
Oct 30 21:29:08.410413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.410294    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca\") pod \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") "
Oct 30 21:29:08.410413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.410326    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca\") pod \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\" (UID: \"47bf0bd5-3d72-48ef-89f0-45f2efea3955\") "
Oct 30 21:29:08.410917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.410887    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca" pod "47bf0bd5-3d72-48ef-89f0-45f2efea3955" (UID: "47bf0bd5-3d72-48ef-89f0-45f2efea3955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:08.417565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.417534    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn" pod "47bf0bd5-3d72-48ef-89f0-45f2efea3955" (UID: "47bf0bd5-3d72-48ef-89f0-45f2efea3955"). InnerVolumeSpecName "kube-api-access-5cpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:08.418863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.418837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca" pod "47bf0bd5-3d72-48ef-89f0-45f2efea3955" (UID: "47bf0bd5-3d72-48ef-89f0-45f2efea3955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:08.512371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.512333    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpgn\" (UniqueName: \"kubernetes.io/projected/47bf0bd5-3d72-48ef-89f0-45f2efea3955-kube-api-access-5cpgn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:08.512371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.512376    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:08.512577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.512391    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf0bd5-3d72-48ef-89f0-45f2efea3955-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:08.718674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.716664    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"]
Oct 30 21:29:08.718674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.717820    6695 cpu_manager.go:414] "containerMap: removing container" podUID="47bf0bd5-3d72-48ef-89f0-45f2efea3955" containerName="revoke-keys"
Oct 30 21:29:08.718674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.717848    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47bf0bd5-3d72-48ef-89f0-45f2efea3955" containerName="revoke-keys"
Oct 30 21:29:08.718674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.718009    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="47bf0bd5-3d72-48ef-89f0-45f2efea3955" containerName="revoke-keys"
Oct 30 21:29:08.727625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.722627    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:08.752420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.746506    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ab5774\""
Oct 30 21:29:08.752420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.747257    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ab5774-dockercfg-77669\""
Oct 30 21:29:08.813835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.813520    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"]
Oct 30 21:29:08.879680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.865276    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:08.879680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.865361    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqrl\" (UniqueName: \"kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl\") pod \"payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:08.966381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.966342    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:08.966569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.966407    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqrl\" (UniqueName: \"kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl\") pod \"payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:08.967353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.967323    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:08.986911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:08.986835    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqrl\" (UniqueName: \"kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl\") pod \"payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") " pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:09.179561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.179267    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:09.202763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.202724    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97" event={"ID":"47bf0bd5-3d72-48ef-89f0-45f2efea3955","Type":"ContainerDied","Data":"8179207a8c7e70c75181ff1a21fccbc20925d6111e28df8a405d25d3761743ee"}
Oct 30 21:29:09.202991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.202972    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8179207a8c7e70c75181ff1a21fccbc20925d6111e28df8a405d25d3761743ee"
Oct 30 21:29:09.203162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.203150    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"
Oct 30 21:29:09.723097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.723062    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"]
Oct 30 21:29:09.734801 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:29:09.730030    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15cff4a_47a0_4537_b21b_5c556b18120e.slice/crio-ed1fcd90e04d921489264117da277e719789a700d213ff23bf7b51b1e60fc66a WatchSource:0}: Error finding container ed1fcd90e04d921489264117da277e719789a700d213ff23bf7b51b1e60fc66a: Status 404 returned error can't find the container with id ed1fcd90e04d921489264117da277e719789a700d213ff23bf7b51b1e60fc66a
Oct 30 21:29:09.834820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.834796    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:29:09.914359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914323    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.914592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914463    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.914592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914523    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914732    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914865    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914933    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.914996    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915052    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915101    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs\") pod \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\" (UID: \"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b\") "
Oct 30 21:29:09.915470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915439    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.915530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915460    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:09.915694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915672    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.915767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915726    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.915971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915950    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.915971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915971    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.916136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915985    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.916136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.915997    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:09.916445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.916418    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.917669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.917638    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "kube-api-access-bgmgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:09.917959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.917936    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:09.918122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.918095    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:09.918248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:09.918230    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo" pod "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" (UID: "6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:29:10.017483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.017382    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:10.017483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.017418    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:10.017483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.017433    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:10.017483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.017443    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:10.017483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.017455    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmgg\" (UniqueName: \"kubernetes.io/projected/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b-kube-api-access-bgmgg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:10.212521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.212475    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv" event={"ID":"c15cff4a-47a0-4537-b21b-5c556b18120e","Type":"ContainerStarted","Data":"5f9179c54fb7bbb1bbe5e11667e87f84cf8e95033f45b4fe8025afcdda256574"}
Oct 30 21:29:10.212758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.212532    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv" event={"ID":"c15cff4a-47a0-4537-b21b-5c556b18120e","Type":"ContainerStarted","Data":"ed1fcd90e04d921489264117da277e719789a700d213ff23bf7b51b1e60fc66a"}
Oct 30 21:29:10.212758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.212711    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:10.214285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.214260    6695 generic.go:356] "Generic (PLEG): container finished" podID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerID="4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698" exitCode=0
Oct 30 21:29:10.214389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.214304    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"
Oct 30 21:29:10.214389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.214339    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" event={"ID":"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b","Type":"ContainerDied","Data":"4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698"}
Oct 30 21:29:10.214459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.214387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx" event={"ID":"6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b","Type":"ContainerDied","Data":"fef02cb826079fc0d523ad2d929091c1478aedc3d3421798dbcf0ff4c9715800"}
Oct 30 21:29:10.214459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.214412    6695 scope.go:122] "RemoveContainer" containerID="4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698"
Oct 30 21:29:10.226157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.225912    6695 scope.go:122] "RemoveContainer" containerID="617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"
Oct 30 21:29:10.235736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.235717    6695 scope.go:122] "RemoveContainer" containerID="4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698"
Oct 30 21:29:10.236006 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:10.235982    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698\": container with ID starting with 4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698 not found: ID does not exist" containerID="4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698"
Oct 30 21:29:10.236055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.236015    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698"} err="failed to get container status \"4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698\": rpc error: code = NotFound desc = could not find container \"4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698\": container with ID starting with 4048be9c31a846b7292e1e628a636454a772485b9ffdca894294295c53db0698 not found: ID does not exist"
Oct 30 21:29:10.236055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.236032    6695 scope.go:122] "RemoveContainer" containerID="617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"
Oct 30 21:29:10.236269 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:10.236249    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f\": container with ID starting with 617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f not found: ID does not exist" containerID="617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"
Oct 30 21:29:10.236313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.236275    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f"} err="failed to get container status \"617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f\": rpc error: code = NotFound desc = could not find container \"617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f\": container with ID starting with 617e24aa247b516f417fa0c5bc238f2246f35e1ac1c80465c72611992eaf104f not found: ID does not exist"
Oct 30 21:29:10.260842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.258460    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv" podStartSLOduration=2.25844486 podStartE2EDuration="2.25844486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:10.257233471 +0000 UTC m=+1133.179691770" watchObservedRunningTime="2026-10-30 21:29:10.25844486 +0000 UTC m=+1133.180903182"
Oct 30 21:29:10.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.300790    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"]
Oct 30 21:29:10.317900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:10.316848    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-659efa-openshift-default-77d59999bf-gfkfx"]
Oct 30 21:29:11.215251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:11.215216    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" path="/var/lib/kubelet/pods/6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b/volumes"
Oct 30 21:29:15.372713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:15.372674    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"]
Oct 30 21:29:15.409842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:15.409808    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"]
Oct 30 21:29:17.879701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.879667    6695 scope.go:122] "RemoveContainer" containerID="b25ce73f472bf6be8ec83be98c554136dfd50175b19f48a372982a24bf00a38f"
Oct 30 21:29:17.891437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.891415    6695 scope.go:122] "RemoveContainer" containerID="fe69f80a8f64d6e6c3e20f26599dc9dc97cda7758688ffb05af72a7e64d1e054"
Oct 30 21:29:17.901817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.901798    6695 scope.go:122] "RemoveContainer" containerID="e19d9e02e06733779528ee88c185ea06cf3153e3fd2c847fbd8c4943148378bc"
Oct 30 21:29:17.911788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.911769    6695 scope.go:122] "RemoveContainer" containerID="3f72f09aab445d3f87c049cf08757e1335c38d240b9efcafe0d24484781cf47c"
Oct 30 21:29:17.921724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.921703    6695 scope.go:122] "RemoveContainer" containerID="7e5f783fe276c853f00da45997960a64ef2f9983a19ec94c8580d9d383540c64"
Oct 30 21:29:17.931204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.931180    6695 scope.go:122] "RemoveContainer" containerID="5d1903a5cd0fa9229fc0d6300590f6fdfc6e92549cc19e873709bca6d17ce532"
Oct 30 21:29:17.943570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:17.943547    6695 scope.go:122] "RemoveContainer" containerID="ac859775beecee786a8a734d3e16380d5ecbd6dd082b9ac42988c583dfbaf286"
Oct 30 21:29:18.282834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.282797    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"]
Oct 30 21:29:18.283510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.283494    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy"
Oct 30 21:29:18.283566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.283512    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy"
Oct 30 21:29:18.283566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.283546    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy"
Oct 30 21:29:18.283566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.283552    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy"
Oct 30 21:29:18.283710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.283673    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy"
Oct 30 21:29:18.289833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.289798    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.307742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.307703    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.307928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.307906    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjjc\" (UniqueName: \"kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.308145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.308118    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.309528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.309463    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"]
Oct 30 21:29:18.324913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.324886    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:18.325114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.325090    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:29:18.325198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.325184    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:18.325260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.325243    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:18.409342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.409301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjjc\" (UniqueName: \"kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.409519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.409433    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.409519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.409494    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.410202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.410179    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.411885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.411857    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.428616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.428586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjjc\" (UniqueName: \"kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc\") pod \"maas-api-revoke-sub-b635b941-d548v\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.610509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.610430    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:18.916979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:18.916945    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"]
Oct 30 21:29:18.919518 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:29:18.919491    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a50edf_5f68_4ef2_9eeb_b60ee63abb18.slice/crio-b197977db749075daeaeed7415c5b02293ccebd3e791510b072d2abbda61dbb4 WatchSource:0}: Error finding container b197977db749075daeaeed7415c5b02293ccebd3e791510b072d2abbda61dbb4: Status 404 returned error can't find the container with id b197977db749075daeaeed7415c5b02293ccebd3e791510b072d2abbda61dbb4
Oct 30 21:29:19.302709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:19.302669    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v" event={"ID":"45a50edf-5f68-4ef2-9eeb-b60ee63abb18","Type":"ContainerStarted","Data":"b197977db749075daeaeed7415c5b02293ccebd3e791510b072d2abbda61dbb4"}
Oct 30 21:29:20.313232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.313196    6695 generic.go:356] "Generic (PLEG): container finished" podID="45a50edf-5f68-4ef2-9eeb-b60ee63abb18" containerID="8080418b1cedbe04cbdbecc795937a13cb835ce469948ac13900b7d3038bf24b" exitCode=0
Oct 30 21:29:20.313682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.313252    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v" event={"ID":"45a50edf-5f68-4ef2-9eeb-b60ee63abb18","Type":"ContainerDied","Data":"8080418b1cedbe04cbdbecc795937a13cb835ce469948ac13900b7d3038bf24b"}
Oct 30 21:29:20.417634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.402616    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r" podUID="86f2e646-7a58-426a-b395-a7102953651d" containerName="payload-processing" containerID="cri-o://2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5" gracePeriod=25
Oct 30 21:29:20.419090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.418368    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz" podUID="e140e7b1-469a-4872-83a1-8327f9ebfa76" containerName="payload-pre-processing" containerID="cri-o://164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593" gracePeriod=25
Oct 30 21:29:20.747667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.747597    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:29:20.774660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.774637    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:29:20.852323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.852279    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k\" (UniqueName: \"kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k\") pod \"86f2e646-7a58-426a-b395-a7102953651d\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") "
Oct 30 21:29:20.852525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.852334    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume\") pod \"86f2e646-7a58-426a-b395-a7102953651d\" (UID: \"86f2e646-7a58-426a-b395-a7102953651d\") "
Oct 30 21:29:20.852525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.852379    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume\") pod \"e140e7b1-469a-4872-83a1-8327f9ebfa76\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") "
Oct 30 21:29:20.852719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.852567    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx\" (UniqueName: \"kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx\") pod \"e140e7b1-469a-4872-83a1-8327f9ebfa76\" (UID: \"e140e7b1-469a-4872-83a1-8327f9ebfa76\") "
Oct 30 21:29:20.852817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.852772    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume" pod "e140e7b1-469a-4872-83a1-8327f9ebfa76" (UID: "e140e7b1-469a-4872-83a1-8327f9ebfa76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:20.852817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.852771    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume" pod "86f2e646-7a58-426a-b395-a7102953651d" (UID: "86f2e646-7a58-426a-b395-a7102953651d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:20.853545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.853517    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f2e646-7a58-426a-b395-a7102953651d-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:20.853693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.853550    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e140e7b1-469a-4872-83a1-8327f9ebfa76-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:20.855909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.855877    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx" pod "e140e7b1-469a-4872-83a1-8327f9ebfa76" (UID: "e140e7b1-469a-4872-83a1-8327f9ebfa76"). InnerVolumeSpecName "kube-api-access-lrjvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:20.856339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.856314    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k" pod "86f2e646-7a58-426a-b395-a7102953651d" (UID: "86f2e646-7a58-426a-b395-a7102953651d"). InnerVolumeSpecName "kube-api-access-kl98k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:20.964288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.961197    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl98k\" (UniqueName: \"kubernetes.io/projected/86f2e646-7a58-426a-b395-a7102953651d-kube-api-access-kl98k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:20.964288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:20.961239    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjvx\" (UniqueName: \"kubernetes.io/projected/e140e7b1-469a-4872-83a1-8327f9ebfa76-kube-api-access-lrjvx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:21.226538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.226247    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:29:21.324926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.324893    6695 generic.go:356] "Generic (PLEG): container finished" podID="e140e7b1-469a-4872-83a1-8327f9ebfa76" containerID="164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593" exitCode=0
Oct 30 21:29:21.325361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.324933    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"
Oct 30 21:29:21.325361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.324979    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz" event={"ID":"e140e7b1-469a-4872-83a1-8327f9ebfa76","Type":"ContainerDied","Data":"164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593"}
Oct 30 21:29:21.325361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.325045    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz" event={"ID":"e140e7b1-469a-4872-83a1-8327f9ebfa76","Type":"ContainerDied","Data":"45a6248760b7b90cfd6de8e0a352678c68d05ebfa4bdc16903c3b1bd7b5ab2e1"}
Oct 30 21:29:21.325361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.325074    6695 scope.go:122] "RemoveContainer" containerID="164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593"
Oct 30 21:29:21.327592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.327564    6695 generic.go:356] "Generic (PLEG): container finished" podID="86f2e646-7a58-426a-b395-a7102953651d" containerID="2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5" exitCode=0
Oct 30 21:29:21.327729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.327643    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"
Oct 30 21:29:21.327729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.327680    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r" event={"ID":"86f2e646-7a58-426a-b395-a7102953651d","Type":"ContainerDied","Data":"2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5"}
Oct 30 21:29:21.330039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.330011    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r" event={"ID":"86f2e646-7a58-426a-b395-a7102953651d","Type":"ContainerDied","Data":"e2c3a88ff835b53b54d8806211ac7e2a99df5e4548037fc3637d8330d9a45c99"}
Oct 30 21:29:21.336670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.336643    6695 scope.go:122] "RemoveContainer" containerID="164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593"
Oct 30 21:29:21.337050 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:21.337026    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593\": container with ID starting with 164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593 not found: ID does not exist" containerID="164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593"
Oct 30 21:29:21.337123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.337064    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593"} err="failed to get container status \"164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593\": rpc error: code = NotFound desc = could not find container \"164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593\": container with ID starting with 164df2e4a57f5c26287558440fcaa096942e681261626b228823282cb5cab593 not found: ID does not exist"
Oct 30 21:29:21.337123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.337088    6695 scope.go:122] "RemoveContainer" containerID="2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5"
Oct 30 21:29:21.350495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.350469    6695 scope.go:122] "RemoveContainer" containerID="2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5"
Oct 30 21:29:21.350875 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:21.350851    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5\": container with ID starting with 2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5 not found: ID does not exist" containerID="2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5"
Oct 30 21:29:21.350978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.350885    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5"} err="failed to get container status \"2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5\": rpc error: code = NotFound desc = could not find container \"2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5\": container with ID starting with 2cb265c9069a34a37f39df2ae44cdd1e8591e07e7598bec8a9bcc3b8d24e5ac5 not found: ID does not exist"
Oct 30 21:29:21.376704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.376663    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"]
Oct 30 21:29:21.383082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.383053    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-834924-5b4b79dc4f-vzb6r"]
Oct 30 21:29:21.418661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.417571    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"]
Oct 30 21:29:21.422486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.422425    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-834924-56bbd98dc-r2chz"]
Oct 30 21:29:21.516237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.516211    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:21.574416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.574381    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca\") pod \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") "
Oct 30 21:29:21.574599 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.574578    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc\" (UniqueName: \"kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc\") pod \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") "
Oct 30 21:29:21.574733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.574712    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca\") pod \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\" (UID: \"45a50edf-5f68-4ef2-9eeb-b60ee63abb18\") "
Oct 30 21:29:21.575452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.575135    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca" pod "45a50edf-5f68-4ef2-9eeb-b60ee63abb18" (UID: "45a50edf-5f68-4ef2-9eeb-b60ee63abb18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:21.575452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.575422    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:21.576997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.576966    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc" pod "45a50edf-5f68-4ef2-9eeb-b60ee63abb18" (UID: "45a50edf-5f68-4ef2-9eeb-b60ee63abb18"). InnerVolumeSpecName "kube-api-access-8fjjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:21.577112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.577016    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca" pod "45a50edf-5f68-4ef2-9eeb-b60ee63abb18" (UID: "45a50edf-5f68-4ef2-9eeb-b60ee63abb18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:21.676719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.676674    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjjc\" (UniqueName: \"kubernetes.io/projected/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-kube-api-access-8fjjc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:21.676719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:21.676715    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a50edf-5f68-4ef2-9eeb-b60ee63abb18-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:22.336231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:22.336200    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"
Oct 30 21:29:22.336708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:22.336199    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v" event={"ID":"45a50edf-5f68-4ef2-9eeb-b60ee63abb18","Type":"ContainerDied","Data":"b197977db749075daeaeed7415c5b02293ccebd3e791510b072d2abbda61dbb4"}
Oct 30 21:29:22.336708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:22.336357    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b197977db749075daeaeed7415c5b02293ccebd3e791510b072d2abbda61dbb4"
Oct 30 21:29:23.191963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.191923    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"]
Oct 30 21:29:23.192677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192660    6695 cpu_manager.go:414] "containerMap: removing container" podUID="86f2e646-7a58-426a-b395-a7102953651d" containerName="payload-processing"
Oct 30 21:29:23.192731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192678    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f2e646-7a58-426a-b395-a7102953651d" containerName="payload-processing"
Oct 30 21:29:23.192731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192693    6695 cpu_manager.go:414] "containerMap: removing container" podUID="45a50edf-5f68-4ef2-9eeb-b60ee63abb18" containerName="revoke-keys"
Oct 30 21:29:23.192731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192698    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a50edf-5f68-4ef2-9eeb-b60ee63abb18" containerName="revoke-keys"
Oct 30 21:29:23.192731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192729    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e140e7b1-469a-4872-83a1-8327f9ebfa76" containerName="payload-pre-processing"
Oct 30 21:29:23.192906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192734    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e140e7b1-469a-4872-83a1-8327f9ebfa76" containerName="payload-pre-processing"
Oct 30 21:29:23.192906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192867    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e140e7b1-469a-4872-83a1-8327f9ebfa76" containerName="payload-pre-processing"
Oct 30 21:29:23.192906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192883    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f2e646-7a58-426a-b395-a7102953651d" containerName="payload-processing"
Oct 30 21:29:23.192906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192890    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a50edf-5f68-4ef2-9eeb-b60ee63abb18" containerName="revoke-keys"
Oct 30 21:29:23.192906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.192899    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3ab0c9-4321-41d1-a2b2-771cdb5dd53b" containerName="istio-proxy"
Oct 30 21:29:23.199346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.199323    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.218982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.218948    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f2e646-7a58-426a-b395-a7102953651d" path="/var/lib/kubelet/pods/86f2e646-7a58-426a-b395-a7102953651d/volumes"
Oct 30 21:29:23.219577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.219558    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e140e7b1-469a-4872-83a1-8327f9ebfa76" path="/var/lib/kubelet/pods/e140e7b1-469a-4872-83a1-8327f9ebfa76/volumes"
Oct 30 21:29:23.219926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.219911    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"]
Oct 30 21:29:23.221480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.221437    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:23.221480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.221445    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:23.221480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.221446    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:23.225092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.225069    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:29:23.281193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.281157    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.281406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.281391    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495jn\" (UniqueName: \"kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.281503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.281491    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.382820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.382784    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.383293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.382830    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495jn\" (UniqueName: \"kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.383293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.382930    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.383584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.383559    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.385314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.385287    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.420273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.420223    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495jn\" (UniqueName: \"kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn\") pod \"maas-api-revoke-keys-e2e-api-a-b13793-nfb96\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.515897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.515809    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:23.832068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:23.832032    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"]
Oct 30 21:29:23.835460 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:29:23.835424    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486c6502_1225_41d4_b4d6_b4d4d3d8a791.slice/crio-405a267f701cdd46cdd91bdaee462e67dc598d7e9c399a05538f41b39e6aca92 WatchSource:0}: Error finding container 405a267f701cdd46cdd91bdaee462e67dc598d7e9c399a05538f41b39e6aca92: Status 404 returned error can't find the container with id 405a267f701cdd46cdd91bdaee462e67dc598d7e9c399a05538f41b39e6aca92
Oct 30 21:29:24.354836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:24.354797    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96" event={"ID":"486c6502-1225-41d4-b4d6-b4d4d3d8a791","Type":"ContainerStarted","Data":"197f917e2c71a0b222b5825d7e06b612e7aa81edcd9d4936c72ff037e5cc4b06"}
Oct 30 21:29:24.355020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:24.354857    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96" event={"ID":"486c6502-1225-41d4-b4d6-b4d4d3d8a791","Type":"ContainerStarted","Data":"405a267f701cdd46cdd91bdaee462e67dc598d7e9c399a05538f41b39e6aca92"}
Oct 30 21:29:25.364984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:25.364952    6695 generic.go:356] "Generic (PLEG): container finished" podID="486c6502-1225-41d4-b4d6-b4d4d3d8a791" containerID="197f917e2c71a0b222b5825d7e06b612e7aa81edcd9d4936c72ff037e5cc4b06" exitCode=0
Oct 30 21:29:25.365414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:25.365038    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96" event={"ID":"486c6502-1225-41d4-b4d6-b4d4d3d8a791","Type":"ContainerDied","Data":"197f917e2c71a0b222b5825d7e06b612e7aa81edcd9d4936c72ff037e5cc4b06"}
Oct 30 21:29:26.543509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.543483    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:26.649393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.649294    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca\") pod \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") "
Oct 30 21:29:26.649558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.649491    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn\" (UniqueName: \"kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn\") pod \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") "
Oct 30 21:29:26.649558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.649538    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca\") pod \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\" (UID: \"486c6502-1225-41d4-b4d6-b4d4d3d8a791\") "
Oct 30 21:29:26.649753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.649722    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca" pod "486c6502-1225-41d4-b4d6-b4d4d3d8a791" (UID: "486c6502-1225-41d4-b4d6-b4d4d3d8a791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:26.650126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.650102    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:26.651676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.651618    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn" pod "486c6502-1225-41d4-b4d6-b4d4d3d8a791" (UID: "486c6502-1225-41d4-b4d6-b4d4d3d8a791"). InnerVolumeSpecName "kube-api-access-495jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:26.651676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.651596    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca" pod "486c6502-1225-41d4-b4d6-b4d4d3d8a791" (UID: "486c6502-1225-41d4-b4d6-b4d4d3d8a791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:26.751130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.751091    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-495jn\" (UniqueName: \"kubernetes.io/projected/486c6502-1225-41d4-b4d6-b4d4d3d8a791-kube-api-access-495jn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:26.751130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:26.751121    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486c6502-1225-41d4-b4d6-b4d4d3d8a791-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:27.385982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:27.385950    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"
Oct 30 21:29:27.386135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:27.385980    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96" event={"ID":"486c6502-1225-41d4-b4d6-b4d4d3d8a791","Type":"ContainerDied","Data":"405a267f701cdd46cdd91bdaee462e67dc598d7e9c399a05538f41b39e6aca92"}
Oct 30 21:29:27.386135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:27.386037    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405a267f701cdd46cdd91bdaee462e67dc598d7e9c399a05538f41b39e6aca92"
Oct 30 21:29:31.073796 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:31.073760    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:29:33.783952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:33.783912    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"]
Oct 30 21:29:33.839073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:33.839029    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"]
Oct 30 21:29:34.328644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.324799    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:34.592690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.571814    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:34.730100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.730052    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gxjrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:29:34.730418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.730368    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:34.730835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.730805    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:34.787629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.784372    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-xhqfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:29:34.787629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:34.784484    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:35.056787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.056743    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:35.133834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.133764    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:35.342181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.342080    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:29:35.342825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.342796    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:29:35.628853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.628710    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 30 21:29:35.742114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.734712    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:35.786703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.786666    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:29:35.828626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.827388    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:29:35.900998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:35.900920    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:29:38.785184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:38.785139    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d" podUID="c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" containerName="payload-processing" containerID="cri-o://761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69" gracePeriod=25
Oct 30 21:29:38.866008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:38.865967    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47" podUID="5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" containerName="payload-pre-processing" containerID="cri-o://517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13" gracePeriod=25
Oct 30 21:29:39.191654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.190814    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:29:39.270360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.270335    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:29:39.285945 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.285917    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc\" (UniqueName: \"kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc\") pod \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") "
Oct 30 21:29:39.286148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.285959    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume\") pod \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\" (UID: \"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67\") "
Oct 30 21:29:39.286683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.286394    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume" pod "c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" (UID: "c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:39.286812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.286787    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:39.288323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.288298    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc" pod "c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" (UID: "c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67"). InnerVolumeSpecName "kube-api-access-xdjjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:39.387528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.387436    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume\") pod \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") "
Oct 30 21:29:39.387700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.387661    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht\" (UniqueName: \"kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht\") pod \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\" (UID: \"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8\") "
Oct 30 21:29:39.387840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.387815    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume" pod "5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" (UID: "5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:39.388465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.388444    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjjc\" (UniqueName: \"kubernetes.io/projected/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67-kube-api-access-xdjjc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:39.388511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.388473    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:39.389637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.389586    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht" pod "5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" (UID: "5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8"). InnerVolumeSpecName "kube-api-access-dhwht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:39.489835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.489800    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwht\" (UniqueName: \"kubernetes.io/projected/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8-kube-api-access-dhwht\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:29:39.842570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.842536    6695 generic.go:356] "Generic (PLEG): container finished" podID="5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" containerID="517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13" exitCode=0
Oct 30 21:29:39.842570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.842582    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"
Oct 30 21:29:39.843142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.842653    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47" event={"ID":"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8","Type":"ContainerDied","Data":"517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13"}
Oct 30 21:29:39.843142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.842743    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47" event={"ID":"5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8","Type":"ContainerDied","Data":"d81ccaeae490ec64fe48620c8dd2c4c0b6e7d2a9de55cc29fc5b29745b3d76cd"}
Oct 30 21:29:39.843142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.842775    6695 scope.go:122] "RemoveContainer" containerID="517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13"
Oct 30 21:29:39.844551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.844530    6695 generic.go:356] "Generic (PLEG): container finished" podID="c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" containerID="761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69" exitCode=0
Oct 30 21:29:39.844715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.844581    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"
Oct 30 21:29:39.844715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.844673    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d" event={"ID":"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67","Type":"ContainerDied","Data":"761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69"}
Oct 30 21:29:39.844844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.844750    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d" event={"ID":"c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67","Type":"ContainerDied","Data":"c26887e061e7aad1552abed67d5ecc6167948b61226a4dc79abad6c9fef158e6"}
Oct 30 21:29:39.855435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.855400    6695 scope.go:122] "RemoveContainer" containerID="517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13"
Oct 30 21:29:39.855759 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:39.855739    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13\": container with ID starting with 517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13 not found: ID does not exist" containerID="517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13"
Oct 30 21:29:39.855861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.855765    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13"} err="failed to get container status \"517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13\": rpc error: code = NotFound desc = could not find container \"517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13\": container with ID starting with 517ff4761ae2f86be139f54d678558fbdcbf9df910f1a3675b6b2dc4d6978c13 not found: ID does not exist"
Oct 30 21:29:39.855861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.855787    6695 scope.go:122] "RemoveContainer" containerID="761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69"
Oct 30 21:29:39.866205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.866123    6695 scope.go:122] "RemoveContainer" containerID="761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69"
Oct 30 21:29:39.866468 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:29:39.866446    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69\": container with ID starting with 761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69 not found: ID does not exist" containerID="761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69"
Oct 30 21:29:39.866579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.866473    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69"} err="failed to get container status \"761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69\": rpc error: code = NotFound desc = could not find container \"761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69\": container with ID starting with 761a3286e7a528dd924e48d9c797f330c2ac1d93486fc0740bc100a5d3e44e69 not found: ID does not exist"
Oct 30 21:29:39.908680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.907844    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"]
Oct 30 21:29:39.951206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.951169    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b13793-6d7cc45cb5-dpv4d"]
Oct 30 21:29:39.972951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.972908    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"]
Oct 30 21:29:39.979770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:39.979741    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b13793-84c6bcb6b7-str47"]
Oct 30 21:29:41.068823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:41.068775    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"]
Oct 30 21:29:41.078287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:41.078253    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-c7z2t"]
Oct 30 21:29:41.239656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:41.234576    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" path="/var/lib/kubelet/pods/5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8/volumes"
Oct 30 21:29:41.239656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:41.235347    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd07fc9-3fe5-452b-9832-e5065e597e8b" path="/var/lib/kubelet/pods/9dd07fc9-3fe5-452b-9832-e5065e597e8b/volumes"
Oct 30 21:29:41.239656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:41.236839    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" path="/var/lib/kubelet/pods/c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67/volumes"
Oct 30 21:29:43.126934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.126891    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"]
Oct 30 21:29:43.127978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.127943    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" containerName="payload-processing"
Oct 30 21:29:43.127978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.127970    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" containerName="payload-processing"
Oct 30 21:29:43.128175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.127990    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" containerName="payload-pre-processing"
Oct 30 21:29:43.128175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.127998    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" containerName="payload-pre-processing"
Oct 30 21:29:43.128175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.128024    6695 cpu_manager.go:414] "containerMap: removing container" podUID="486c6502-1225-41d4-b4d6-b4d4d3d8a791" containerName="revoke-keys"
Oct 30 21:29:43.128175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.128033    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486c6502-1225-41d4-b4d6-b4d4d3d8a791" containerName="revoke-keys"
Oct 30 21:29:43.128385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.128255    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51f0fa4-9e4f-44fb-a9a5-6ccb0dee4d67" containerName="payload-processing"
Oct 30 21:29:43.128385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.128273    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfd29ad-1c1b-47a9-a6da-c81cf3cd71b8" containerName="payload-pre-processing"
Oct 30 21:29:43.128385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.128285    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="486c6502-1225-41d4-b4d6-b4d4d3d8a791" containerName="revoke-keys"
Oct 30 21:29:43.134230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.134197    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.157188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.157156    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"]
Oct 30 21:29:43.158184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.158156    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-342fc9-openshift-default-dockercfg-xbntp\""
Oct 30 21:29:43.257332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257289    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257360    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257384    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257422    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257575    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257597    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lnp\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257695    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257748    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.257969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.257836    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.359626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.359497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.359820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.359664    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.359820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.359742    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.359820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.359765    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.359820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.359818    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.360073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.360015    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.360073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.360046    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lnp\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.360189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.360102    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.360189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.360132    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.370544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.370511    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.378744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.371152    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.388727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.374571    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.388962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.379985    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.389063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.380228    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.389147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.380460    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.442929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.442894    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.458647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.446150    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.491358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.491314    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lnp\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp\") pod \"e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") " pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:43.761244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:43.761146    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:44.090173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:44.090141    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"]
Oct 30 21:29:44.092325 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:29:44.092298    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4cd3c8_25a0_477a_976c_b591bb067e68.slice/crio-556913c1c1f5e8ad5b270b4a7992525a01dc84c3169c72cf44543fcb30491051 WatchSource:0}: Error finding container 556913c1c1f5e8ad5b270b4a7992525a01dc84c3169c72cf44543fcb30491051: Status 404 returned error can't find the container with id 556913c1c1f5e8ad5b270b4a7992525a01dc84c3169c72cf44543fcb30491051
Oct 30 21:29:44.094730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:44.094698    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:44.094822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:44.094765    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:44.094822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:44.094794    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:44.121765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:44.121733    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" event={"ID":"8a4cd3c8-25a0-477a-976c-b591bb067e68","Type":"ContainerStarted","Data":"556913c1c1f5e8ad5b270b4a7992525a01dc84c3169c72cf44543fcb30491051"}
Oct 30 21:29:45.130574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:45.130534    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" event={"ID":"8a4cd3c8-25a0-477a-976c-b591bb067e68","Type":"ContainerStarted","Data":"55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"}
Oct 30 21:29:45.196008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:45.195952    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podStartSLOduration=2.195935178 podStartE2EDuration="2.195935178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:45.181169409 +0000 UTC m=+1168.103627710" watchObservedRunningTime="2026-10-30 21:29:45.195935178 +0000 UTC m=+1168.118393478"
Oct 30 21:29:45.761373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:45.761339    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:45.766903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:45.766875    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:46.142472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.142437    6695 generic.go:356] "Generic (PLEG): container finished" podID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerID="586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6" exitCode=0
Oct 30 21:29:46.142933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.142506    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" event={"ID":"f3f502ae-377f-48ef-b68f-21d116b0d87c","Type":"ContainerDied","Data":"586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"}
Oct 30 21:29:46.143026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.143008    6695 scope.go:122] "RemoveContainer" containerID="586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"
Oct 30 21:29:46.143136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.143120    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:46.144147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.144130    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:29:46.144723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.144695    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:46.144783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.144769    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:46.144836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:46.144805    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:29:47.154023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:47.153987    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" event={"ID":"f3f502ae-377f-48ef-b68f-21d116b0d87c","Type":"ContainerStarted","Data":"a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042"}
Oct 30 21:29:47.332760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:47.332710    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:47.334414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:47.334210    6695 patch_prober.go:29] interesting pod/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:29:47.334414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:47.334267    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 21:29:47.475681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:47.471691    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"]
Oct 30 21:29:47.481474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:47.481414    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-834924-w7w97"]
Oct 30 21:29:48.337105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:48.337063    6695 patch_prober.go:29] interesting pod/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:29:48.337734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:48.337130    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 21:29:49.238683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:49.238638    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47bf0bd5-3d72-48ef-89f0-45f2efea3955" path="/var/lib/kubelet/pods/47bf0bd5-3d72-48ef-89f0-45f2efea3955/volumes"
Oct 30 21:29:49.320077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:49.320033    6695 patch_prober.go:29] interesting pod/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:29:49.320370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:49.320341    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 21:29:50.119745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:50.119710    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:50.322034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:50.321994    6695 patch_prober.go:29] interesting pod/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:29:50.333673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:50.322266    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 21:29:51.333628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:51.333065    6695 patch_prober.go:29] interesting pod/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:29:51.333628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:51.333130    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 21:29:52.322016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:52.321976    6695 patch_prober.go:29] interesting pod/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:29:52.322314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:52.322279    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 21:29:53.321581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:53.321548    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:53.322632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:53.322593    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:29:54.094492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:54.094191    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"]
Oct 30 21:29:54.102290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:54.102258    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b43f0d-nlkqq"]
Oct 30 21:29:55.216435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:55.216400    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482eeced-b283-4352-b36e-fdbc55e94cb5" path="/var/lib/kubelet/pods/482eeced-b283-4352-b36e-fdbc55e94cb5/volumes"
Oct 30 21:29:55.636988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:55.636939    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"]
Oct 30 21:29:55.637429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:29:55.637395    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy" containerID="cri-o://de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b" gracePeriod=30
Oct 30 21:30:00.153673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.153565    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"]
Oct 30 21:30:00.163102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.163066    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"]
Oct 30 21:30:00.164200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.164173    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.170711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.170681    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"]
Oct 30 21:30:00.173400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.170852    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.184358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.184327    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"]
Oct 30 21:30:00.186268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.185650    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:30:00.186654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.186587    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.217918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.217865    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"]
Oct 30 21:30:00.218453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.218257    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.224674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.224281    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:00.224674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.224399    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:00.242010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.241445    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"]
Oct 30 21:30:00.242010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.241522    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"]
Oct 30 21:30:00.243838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.243804    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9452k\" (UniqueName: \"kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k\") pod \"maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.243956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.243890    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2p4m\" (UniqueName: \"kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.244031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.243962    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-v7rqr\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.244031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.244002    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.244145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.244129    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.244643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.244279    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svn4m\" (UniqueName: \"kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m\") pod \"maas-api-key-cleanup-29889930-v7rqr\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.245680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.244857    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.250664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.249554    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"]
Oct 30 21:30:00.252992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.251700    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.261841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.261816    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"]
Oct 30 21:30:00.262055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.262039    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"]
Oct 30 21:30:00.262182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.262171    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"]
Oct 30 21:30:00.262289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.262279    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"]
Oct 30 21:30:00.262395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.262383    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"]
Oct 30 21:30:00.262582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.262567    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.279591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.279358    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"]
Oct 30 21:30:00.346178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346136    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9452k\" (UniqueName: \"kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k\") pod \"maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.346351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346192    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.346351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346212    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.346351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346253    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2p4m\" (UniqueName: \"kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.346458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346342    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-v7rqr\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.346458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346426    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.346561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346545    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr7q\" (UniqueName: \"kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.346597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346583    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtnw\" (UniqueName: \"kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw\") pod \"maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.346650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346627    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.346820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346803    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.346871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346853    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtc4\" (UniqueName: \"kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.347005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346987    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-v7rqr\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.347060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.346991    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldjw\" (UniqueName: \"kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw\") pod \"maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.347060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.347050    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.347131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.347058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.347131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.347114    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn4m\" (UniqueName: \"kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m\") pod \"maas-api-key-cleanup-29889930-v7rqr\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.347203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.347189    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.365932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.365895    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2p4m\" (UniqueName: \"kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.365932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.365920    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svn4m\" (UniqueName: \"kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m\") pod \"maas-api-key-cleanup-29889930-v7rqr\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.365932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.365907    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9452k\" (UniqueName: \"kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k\") pod \"maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.448644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448513    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtc4\" (UniqueName: \"kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.448644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448593    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldjw\" (UniqueName: \"kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw\") pod \"maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.448901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448667    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.448901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448812    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.448901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448839    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.449066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448952    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr7q\" (UniqueName: \"kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.449066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.448977    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtnw\" (UniqueName: \"kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw\") pod \"maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.449066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.449032    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.449543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.449512    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.449679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.449512    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.449679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.449557    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.449803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.449743    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.472169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.472134    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr7q\" (UniqueName: \"kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.472307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.472131    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldjw\" (UniqueName: \"kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw\") pod \"maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.472649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.472596    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtc4\" (UniqueName: \"kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.474069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.474042    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtnw\" (UniqueName: \"kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw\") pod \"maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.542570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.542535    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:00.569472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.569437    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:00.581986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.581957    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:00.592582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.592374    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:00.599878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.599679    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:00.611691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.608659    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:00.623621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.617667    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:00.885539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.885222    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"]
Oct 30 21:30:00.996782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:00.995810    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:30:01.068104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068067    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.068411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068386    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.068697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068678    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.068811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068723    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.068811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068791    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.068928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068828    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.068928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068913    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.069028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.068983    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.069028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.069017    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket\") pod \"251d9852-8145-4cc5-a9ea-7e295dc460c0\" (UID: \"251d9852-8145-4cc5-a9ea-7e295dc460c0\") "
Oct 30 21:30:01.070687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.070456    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:01.070965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.070938    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:01.071113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.071021    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:01.071231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.071210    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:01.071371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.071351    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:01.111010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.110978    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"]
Oct 30 21:30:01.112269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.112233    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:01.112269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.112257    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "kube-api-access-62z72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:01.112449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.112277    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:01.112449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.112375    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy" pod "251d9852-8145-4cc5-a9ea-7e295dc460c0" (UID: "251d9852-8145-4cc5-a9ea-7e295dc460c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:01.114231 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:30:01.114207    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630efc9b_7c36_4514_b4f1_4a6c1643a7d0.slice/crio-b92d72ff2942aa1ce8479eb0886a2add0410139e6e69927e16c7cfa726ff5e6b WatchSource:0}: Error finding container b92d72ff2942aa1ce8479eb0886a2add0410139e6e69927e16c7cfa726ff5e6b: Status 404 returned error can't find the container with id b92d72ff2942aa1ce8479eb0886a2add0410139e6e69927e16c7cfa726ff5e6b
Oct 30 21:30:01.137437 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:01.137405    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:30:01.143758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.143733    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"]
Oct 30 21:30:01.148847 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:30:01.148804    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd24389a_2e3a_4de2_a26a_6bc05c07e1ce.slice/crio-9a8f229f2e84bb88ab8c0dab8a7b4528c2551f4de72990602bb505fb1f70e223 WatchSource:0}: Error finding container 9a8f229f2e84bb88ab8c0dab8a7b4528c2551f4de72990602bb505fb1f70e223: Status 404 returned error can't find the container with id 9a8f229f2e84bb88ab8c0dab8a7b4528c2551f4de72990602bb505fb1f70e223
Oct 30 21:30:01.171707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171681    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171708    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171723    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171737    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62z72\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-kube-api-access-62z72\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171748    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171760    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251d9852-8145-4cc5-a9ea-7e295dc460c0-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171771    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171780    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251d9852-8145-4cc5-a9ea-7e295dc460c0-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.172009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.171789    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251d9852-8145-4cc5-a9ea-7e295dc460c0-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:01.218878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.218729    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"]
Oct 30 21:30:01.241221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.241190    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"]
Oct 30 21:30:01.255484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.255460    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"]
Oct 30 21:30:01.266316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.266287    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"]
Oct 30 21:30:01.315384 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:30:01.315327    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7d0788_7bd4_4166_9a38_2348592b9cbc.slice/crio-f9bca7545fe3dc3ec4ad88b4563e81ab70724150048b473cb0135f820edbdafd WatchSource:0}: Error finding container f9bca7545fe3dc3ec4ad88b4563e81ab70724150048b473cb0135f820edbdafd: Status 404 returned error can't find the container with id f9bca7545fe3dc3ec4ad88b4563e81ab70724150048b473cb0135f820edbdafd
Oct 30 21:30:01.508540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.508465    6695 generic.go:356] "Generic (PLEG): container finished" podID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerID="de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b" exitCode=0
Oct 30 21:30:01.508540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.508516    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"
Oct 30 21:30:01.508792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.508546    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" event={"ID":"251d9852-8145-4cc5-a9ea-7e295dc460c0","Type":"ContainerDied","Data":"de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b"}
Oct 30 21:30:01.508792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.508598    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd" event={"ID":"251d9852-8145-4cc5-a9ea-7e295dc460c0","Type":"ContainerDied","Data":"16d67a524e904e6ab85f80028487590494f7e39d75785c4ed971cf72dd7264c8"}
Oct 30 21:30:01.508792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.508640    6695 scope.go:122] "RemoveContainer" containerID="de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b"
Oct 30 21:30:01.510165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.510137    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" event={"ID":"689264b2-c504-4124-b951-1c6ef3f05caa","Type":"ContainerStarted","Data":"56600cd843fcd37a95493378e52bd13508a92ecc00b53c5c5c6e97cbfc56d52c"}
Oct 30 21:30:01.512916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.512883    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerStarted","Data":"f9bca7545fe3dc3ec4ad88b4563e81ab70724150048b473cb0135f820edbdafd"}
Oct 30 21:30:01.515143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.515117    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerStarted","Data":"457b26ef0e542248418e2535142d0b649c4e8071a030885f4d1eff314f071f8f"}
Oct 30 21:30:01.516757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.516732    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerStarted","Data":"618a381a77e827db9b8c6a2682b9475615ee4be3b9583342fefd924512d9789c"}
Oct 30 21:30:01.518497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.518474    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerStarted","Data":"9a8f229f2e84bb88ab8c0dab8a7b4528c2551f4de72990602bb505fb1f70e223"}
Oct 30 21:30:01.520084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.520065    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerStarted","Data":"b92d72ff2942aa1ce8479eb0886a2add0410139e6e69927e16c7cfa726ff5e6b"}
Oct 30 21:30:01.522064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.522036    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerStarted","Data":"52e4afd924658c218ae49c6d92a7b90b9520a51f4cd5b04b19d778dfb8f82c61"}
Oct 30 21:30:01.522154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.522069    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerStarted","Data":"44632796795d1fa3f27e56446d66fc729a635bc0f5103549aae2dcc97e1b1183"}
Oct 30 21:30:01.526569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.526549    6695 scope.go:122] "RemoveContainer" containerID="7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"
Oct 30 21:30:01.536840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.536815    6695 scope.go:122] "RemoveContainer" containerID="de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b"
Oct 30 21:30:01.537142 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:01.537123    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b\": container with ID starting with de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b not found: ID does not exist" containerID="de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b"
Oct 30 21:30:01.537196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.537149    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b"} err="failed to get container status \"de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b\": rpc error: code = NotFound desc = could not find container \"de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b\": container with ID starting with de5b808a13bd86ae9c6b963f6e586cb5b152a26738606f433a3c40509b79088b not found: ID does not exist"
Oct 30 21:30:01.537196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.537166    6695 scope.go:122] "RemoveContainer" containerID="7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"
Oct 30 21:30:01.537432 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:01.537416    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf\": container with ID starting with 7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf not found: ID does not exist" containerID="7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"
Oct 30 21:30:01.537479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.537437    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf"} err="failed to get container status \"7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf\": rpc error: code = NotFound desc = could not find container \"7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf\": container with ID starting with 7037eb9566adff90eab78d08cea2462fb8e51ccd96e7e755d8c876d19ce7fbaf not found: ID does not exist"
Oct 30 21:30:01.560510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.560454    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" podStartSLOduration=1.560439998 podStartE2EDuration="1.560439998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.559144671 +0000 UTC m=+1184.481602973" watchObservedRunningTime="2026-10-30 21:30:01.560439998 +0000 UTC m=+1184.482898328"
Oct 30 21:30:01.617381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.616114    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"]
Oct 30 21:30:01.628059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:01.628027    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-834924-openshift-default-5fff9f87d7-tjvrd"]
Oct 30 21:30:02.532714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.532683    6695 generic.go:356] "Generic (PLEG): container finished" podID="689264b2-c504-4124-b951-1c6ef3f05caa" containerID="47132c4877001a7bf3eb83b840578fe5778a972c74a9e8d4133d86a1ad75f7a9" exitCode=60
Oct 30 21:30:02.533116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.532765    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" event={"ID":"689264b2-c504-4124-b951-1c6ef3f05caa","Type":"ContainerDied","Data":"47132c4877001a7bf3eb83b840578fe5778a972c74a9e8d4133d86a1ad75f7a9"}
Oct 30 21:30:02.533116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.533068    6695 scope.go:122] "RemoveContainer" containerID="47132c4877001a7bf3eb83b840578fe5778a972c74a9e8d4133d86a1ad75f7a9"
Oct 30 21:30:02.534749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.534725    6695 generic.go:356] "Generic (PLEG): container finished" podID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerID="4b9d140301ff3ec68b9424888ec86e46c427ba98e70c758341e99fcfbe1928c5" exitCode=60
Oct 30 21:30:02.534862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.534810    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerDied","Data":"4b9d140301ff3ec68b9424888ec86e46c427ba98e70c758341e99fcfbe1928c5"}
Oct 30 21:30:02.535014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.535000    6695 scope.go:122] "RemoveContainer" containerID="4b9d140301ff3ec68b9424888ec86e46c427ba98e70c758341e99fcfbe1928c5"
Oct 30 21:30:02.537257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.537177    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerStarted","Data":"08bb12c0e1f366817e0cd43a5e9fa314a60750eaee739c958e9afd7d136ba6bc"}
Oct 30 21:30:02.537389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.537368    6695 scope.go:122] "RemoveContainer" containerID="08bb12c0e1f366817e0cd43a5e9fa314a60750eaee739c958e9afd7d136ba6bc"
Oct 30 21:30:02.539659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.539561    6695 generic.go:356] "Generic (PLEG): container finished" podID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerID="133e804450cbe95bf75173f3ac757ee3091b99d0b01ec096149520bfeb96a04a" exitCode=60
Oct 30 21:30:02.539785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.539694    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerDied","Data":"133e804450cbe95bf75173f3ac757ee3091b99d0b01ec096149520bfeb96a04a"}
Oct 30 21:30:02.540094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.540076    6695 scope.go:122] "RemoveContainer" containerID="133e804450cbe95bf75173f3ac757ee3091b99d0b01ec096149520bfeb96a04a"
Oct 30 21:30:02.541957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.541938    6695 generic.go:356] "Generic (PLEG): container finished" podID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerID="9ad0bbff7a4c5ad945d948aa2045b3cf6398721e5aa2bc1edd4f9f7a5a9278c5" exitCode=60
Oct 30 21:30:02.542055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.542035    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerDied","Data":"9ad0bbff7a4c5ad945d948aa2045b3cf6398721e5aa2bc1edd4f9f7a5a9278c5"}
Oct 30 21:30:02.542236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.542216    6695 scope.go:122] "RemoveContainer" containerID="9ad0bbff7a4c5ad945d948aa2045b3cf6398721e5aa2bc1edd4f9f7a5a9278c5"
Oct 30 21:30:02.544451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.544142    6695 generic.go:356] "Generic (PLEG): container finished" podID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerID="2e2211cad0c6e1e98bfd7ea0e9320c4bf352a4304b00916c90545a67284d3d8d" exitCode=60
Oct 30 21:30:02.544451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.544289    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerDied","Data":"2e2211cad0c6e1e98bfd7ea0e9320c4bf352a4304b00916c90545a67284d3d8d"}
Oct 30 21:30:02.544972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.544714    6695 scope.go:122] "RemoveContainer" containerID="2e2211cad0c6e1e98bfd7ea0e9320c4bf352a4304b00916c90545a67284d3d8d"
Oct 30 21:30:02.547243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.547143    6695 generic.go:356] "Generic (PLEG): container finished" podID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerID="52e4afd924658c218ae49c6d92a7b90b9520a51f4cd5b04b19d778dfb8f82c61" exitCode=60
Oct 30 21:30:02.547243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.547207    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerDied","Data":"52e4afd924658c218ae49c6d92a7b90b9520a51f4cd5b04b19d778dfb8f82c61"}
Oct 30 21:30:02.547504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:02.547470    6695 scope.go:122] "RemoveContainer" containerID="52e4afd924658c218ae49c6d92a7b90b9520a51f4cd5b04b19d778dfb8f82c61"
Oct 30 21:30:03.218732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.218702    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" path="/var/lib/kubelet/pods/251d9852-8145-4cc5-a9ea-7e295dc460c0/volumes"
Oct 30 21:30:03.558825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.558789    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" event={"ID":"689264b2-c504-4124-b951-1c6ef3f05caa","Type":"ContainerStarted","Data":"27c2c96a7b42cd6c8a899a9c3350f5b2046a4a8cc394ba8441130d09554f1dd8"}
Oct 30 21:30:03.560932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.560901    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerStarted","Data":"fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63"}
Oct 30 21:30:03.562911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.562884    6695 generic.go:356] "Generic (PLEG): container finished" podID="71b8c099-0e21-4843-ac7a-88472df33b27" containerID="08bb12c0e1f366817e0cd43a5e9fa314a60750eaee739c958e9afd7d136ba6bc" exitCode=60
Oct 30 21:30:03.563031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.562966    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerDied","Data":"08bb12c0e1f366817e0cd43a5e9fa314a60750eaee739c958e9afd7d136ba6bc"}
Oct 30 21:30:03.563031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.563014    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerStarted","Data":"72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227"}
Oct 30 21:30:03.564863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.564832    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerStarted","Data":"00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a"}
Oct 30 21:30:03.566775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.566749    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerStarted","Data":"0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17"}
Oct 30 21:30:03.568752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.568729    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerStarted","Data":"389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae"}
Oct 30 21:30:03.570965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.570943    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerStarted","Data":"c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a"}
Oct 30 21:30:03.599145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.599094    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" podStartSLOduration=3.599078838 podStartE2EDuration="3.599078838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.595910253 +0000 UTC m=+1186.518368551" watchObservedRunningTime="2026-10-30 21:30:03.599078838 +0000 UTC m=+1186.521537136"
Oct 30 21:30:03.638457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.638402    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" podStartSLOduration=3.638387216 podStartE2EDuration="3.638387216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.635226998 +0000 UTC m=+1186.557685295" watchObservedRunningTime="2026-10-30 21:30:03.638387216 +0000 UTC m=+1186.560845513"
Oct 30 21:30:03.679365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.679307    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" podStartSLOduration=3.679288492 podStartE2EDuration="3.679288492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.675432057 +0000 UTC m=+1186.597890357" watchObservedRunningTime="2026-10-30 21:30:03.679288492 +0000 UTC m=+1186.601746791"
Oct 30 21:30:03.721695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.721646    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" podStartSLOduration=3.721631151 podStartE2EDuration="3.721631151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.716144526 +0000 UTC m=+1186.638602825" watchObservedRunningTime="2026-10-30 21:30:03.721631151 +0000 UTC m=+1186.644089448"
Oct 30 21:30:03.789980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.789928    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" podStartSLOduration=3.789914734 podStartE2EDuration="3.789914734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.789766391 +0000 UTC m=+1186.712224689" watchObservedRunningTime="2026-10-30 21:30:03.789914734 +0000 UTC m=+1186.712373033"
Oct 30 21:30:03.831900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:03.831844    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" podStartSLOduration=3.831827876 podStartE2EDuration="3.831827876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.826688674 +0000 UTC m=+1186.749146985" watchObservedRunningTime="2026-10-30 21:30:03.831827876 +0000 UTC m=+1186.754286174"
Oct 30 21:30:04.582173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.582131    6695 generic.go:356] "Generic (PLEG): container finished" podID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerID="00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a" exitCode=60
Oct 30 21:30:04.582717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.582209    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerDied","Data":"00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a"}
Oct 30 21:30:04.582717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.582292    6695 scope.go:122] "RemoveContainer" containerID="133e804450cbe95bf75173f3ac757ee3091b99d0b01ec096149520bfeb96a04a"
Oct 30 21:30:04.582717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.582683    6695 scope.go:122] "RemoveContainer" containerID="00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a"
Oct 30 21:30:04.583052 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.583028    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9_odh-ai-gateway-infra(c71aaa5e-9194-4ba2-b3db-fd0f23cbea14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14"
Oct 30 21:30:04.584471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.584449    6695 generic.go:356] "Generic (PLEG): container finished" podID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerID="0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17" exitCode=60
Oct 30 21:30:04.584569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.584512    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerDied","Data":"0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17"}
Oct 30 21:30:04.584836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.584813    6695 scope.go:122] "RemoveContainer" containerID="0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17"
Oct 30 21:30:04.585238 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.585205    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-v7rqr_odh-ai-gateway-infra(cd24389a-2e3a-4de2-a26a-6bc05c07e1ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce"
Oct 30 21:30:04.586503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.586484    6695 generic.go:356] "Generic (PLEG): container finished" podID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerID="389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae" exitCode=60
Oct 30 21:30:04.586588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.586567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerDied","Data":"389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae"}
Oct 30 21:30:04.586873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.586855    6695 scope.go:122] "RemoveContainer" containerID="389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae"
Oct 30 21:30:04.587217 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.587194    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j_odh-ai-gateway-infra(630efc9b-7c36-4514-b4f1-4a6c1643a7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0"
Oct 30 21:30:04.588700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.588655    6695 generic.go:356] "Generic (PLEG): container finished" podID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerID="c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a" exitCode=60
Oct 30 21:30:04.588700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.588694    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerDied","Data":"c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a"}
Oct 30 21:30:04.589025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.589006    6695 scope.go:122] "RemoveContainer" containerID="c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a"
Oct 30 21:30:04.589350 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.589325    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x_odh-ai-gateway-infra(b6db89eb-7b62-4044-808a-8f64364c5eed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed"
Oct 30 21:30:04.591001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.590982    6695 generic.go:356] "Generic (PLEG): container finished" podID="689264b2-c504-4124-b951-1c6ef3f05caa" containerID="27c2c96a7b42cd6c8a899a9c3350f5b2046a4a8cc394ba8441130d09554f1dd8" exitCode=60
Oct 30 21:30:04.591090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.591058    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" event={"ID":"689264b2-c504-4124-b951-1c6ef3f05caa","Type":"ContainerDied","Data":"27c2c96a7b42cd6c8a899a9c3350f5b2046a4a8cc394ba8441130d09554f1dd8"}
Oct 30 21:30:04.591316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.591299    6695 scope.go:122] "RemoveContainer" containerID="27c2c96a7b42cd6c8a899a9c3350f5b2046a4a8cc394ba8441130d09554f1dd8"
Oct 30 21:30:04.591634 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.591598    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw_odh-ai-gateway-infra(689264b2-c504-4124-b951-1c6ef3f05caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" podUID="689264b2-c504-4124-b951-1c6ef3f05caa"
Oct 30 21:30:04.593023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.593000    6695 generic.go:356] "Generic (PLEG): container finished" podID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerID="fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63" exitCode=60
Oct 30 21:30:04.593098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.593067    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerDied","Data":"fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63"}
Oct 30 21:30:04.593300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.593285    6695 scope.go:122] "RemoveContainer" containerID="fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63"
Oct 30 21:30:04.593565 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.593543    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4_odh-ai-gateway-infra(0b7d0788-7bd4-4166-9a38-2348592b9cbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc"
Oct 30 21:30:04.595089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.595067    6695 generic.go:356] "Generic (PLEG): container finished" podID="71b8c099-0e21-4843-ac7a-88472df33b27" containerID="72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227" exitCode=60
Oct 30 21:30:04.595189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.595150    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerDied","Data":"72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227"}
Oct 30 21:30:04.595360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.595345    6695 scope.go:122] "RemoveContainer" containerID="72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227"
Oct 30 21:30:04.595659 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:04.595640    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5_odh-ai-gateway-infra(71b8c099-0e21-4843-ac7a-88472df33b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" podUID="71b8c099-0e21-4843-ac7a-88472df33b27"
Oct 30 21:30:04.598313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.598297    6695 scope.go:122] "RemoveContainer" containerID="9ad0bbff7a4c5ad945d948aa2045b3cf6398721e5aa2bc1edd4f9f7a5a9278c5"
Oct 30 21:30:04.614371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.614341    6695 scope.go:122] "RemoveContainer" containerID="2e2211cad0c6e1e98bfd7ea0e9320c4bf352a4304b00916c90545a67284d3d8d"
Oct 30 21:30:04.631298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.631272    6695 scope.go:122] "RemoveContainer" containerID="52e4afd924658c218ae49c6d92a7b90b9520a51f4cd5b04b19d778dfb8f82c61"
Oct 30 21:30:04.646241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.646219    6695 scope.go:122] "RemoveContainer" containerID="47132c4877001a7bf3eb83b840578fe5778a972c74a9e8d4133d86a1ad75f7a9"
Oct 30 21:30:04.661649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.661575    6695 scope.go:122] "RemoveContainer" containerID="4b9d140301ff3ec68b9424888ec86e46c427ba98e70c758341e99fcfbe1928c5"
Oct 30 21:30:04.676494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:04.676470    6695 scope.go:122] "RemoveContainer" containerID="08bb12c0e1f366817e0cd43a5e9fa314a60750eaee739c958e9afd7d136ba6bc"
Oct 30 21:30:05.612130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.612098    6695 scope.go:122] "RemoveContainer" containerID="27c2c96a7b42cd6c8a899a9c3350f5b2046a4a8cc394ba8441130d09554f1dd8"
Oct 30 21:30:05.612580 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.612514    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw_odh-ai-gateway-infra(689264b2-c504-4124-b951-1c6ef3f05caa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" podUID="689264b2-c504-4124-b951-1c6ef3f05caa"
Oct 30 21:30:05.614280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.614255    6695 scope.go:122] "RemoveContainer" containerID="fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63"
Oct 30 21:30:05.614721 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.614693    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4_odh-ai-gateway-infra(0b7d0788-7bd4-4166-9a38-2348592b9cbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc"
Oct 30 21:30:05.616473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.616448    6695 scope.go:122] "RemoveContainer" containerID="72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227"
Oct 30 21:30:05.616868 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.616833    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5_odh-ai-gateway-infra(71b8c099-0e21-4843-ac7a-88472df33b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" podUID="71b8c099-0e21-4843-ac7a-88472df33b27"
Oct 30 21:30:05.618573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.618549    6695 scope.go:122] "RemoveContainer" containerID="00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a"
Oct 30 21:30:05.618910 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.618880    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9_odh-ai-gateway-infra(c71aaa5e-9194-4ba2-b3db-fd0f23cbea14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14"
Oct 30 21:30:05.620305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.620286    6695 scope.go:122] "RemoveContainer" containerID="0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17"
Oct 30 21:30:05.620574 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.620544    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-v7rqr_odh-ai-gateway-infra(cd24389a-2e3a-4de2-a26a-6bc05c07e1ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce"
Oct 30 21:30:05.622084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.622066    6695 scope.go:122] "RemoveContainer" containerID="389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae"
Oct 30 21:30:05.622397 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.622349    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j_odh-ai-gateway-infra(630efc9b-7c36-4514-b4f1-4a6c1643a7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0"
Oct 30 21:30:05.623911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:05.623891    6695 scope.go:122] "RemoveContainer" containerID="c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a"
Oct 30 21:30:05.624209 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:05.624186    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x_odh-ai-gateway-infra(b6db89eb-7b62-4044-808a-8f64364c5eed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed"
Oct 30 21:30:08.506707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.506666    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"]
Oct 30 21:30:08.784626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.777098    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:08.832659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.830110    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw\" (UniqueName: \"kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw\") pod \"689264b2-c504-4124-b951-1c6ef3f05caa\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") "
Oct 30 21:30:08.832659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.830226    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca\") pod \"689264b2-c504-4124-b951-1c6ef3f05caa\" (UID: \"689264b2-c504-4124-b951-1c6ef3f05caa\") "
Oct 30 21:30:08.832659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.831221    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca" pod "689264b2-c504-4124-b951-1c6ef3f05caa" (UID: "689264b2-c504-4124-b951-1c6ef3f05caa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:08.843633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.842727    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw" pod "689264b2-c504-4124-b951-1c6ef3f05caa" (UID: "689264b2-c504-4124-b951-1c6ef3f05caa"). InnerVolumeSpecName "kube-api-access-2ldjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:08.971643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.968873    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldjw\" (UniqueName: \"kubernetes.io/projected/689264b2-c504-4124-b951-1c6ef3f05caa-kube-api-access-2ldjw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:08.971643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:08.968913    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/689264b2-c504-4124-b951-1c6ef3f05caa-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:09.397182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.397143    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"]
Oct 30 21:30:09.402839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.402809    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b13793-nfb96"]
Oct 30 21:30:09.677954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.677919    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"
Oct 30 21:30:09.677954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.677947    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw" event={"ID":"689264b2-c504-4124-b951-1c6ef3f05caa","Type":"ContainerDied","Data":"56600cd843fcd37a95493378e52bd13508a92ecc00b53c5c5c6e97cbfc56d52c"}
Oct 30 21:30:09.678452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.678007    6695 scope.go:122] "RemoveContainer" containerID="27c2c96a7b42cd6c8a899a9c3350f5b2046a4a8cc394ba8441130d09554f1dd8"
Oct 30 21:30:09.730813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.730776    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"]
Oct 30 21:30:09.735219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:09.735193    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b13793-29889930-79wcw"]
Oct 30 21:30:11.216209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:11.216174    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486c6502-1225-41d4-b4d6-b4d4d3d8a791" path="/var/lib/kubelet/pods/486c6502-1225-41d4-b4d6-b4d4d3d8a791/volumes"
Oct 30 21:30:11.216696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:11.216679    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" path="/var/lib/kubelet/pods/689264b2-c504-4124-b951-1c6ef3f05caa/volumes"
Oct 30 21:30:15.225565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:15.224144    6695 scope.go:122] "RemoveContainer" containerID="0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17"
Oct 30 21:30:15.226840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:15.225490    6695 scope.go:122] "RemoveContainer" containerID="72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227"
Oct 30 21:30:15.228314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:15.228298    6695 scope.go:122] "RemoveContainer" containerID="fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63"
Oct 30 21:30:15.737246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:15.737214    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerStarted","Data":"e57c0d1814d1f1af8a440d828fd83af4a98643c56c462425b05ceb120bbe348b"}
Oct 30 21:30:15.739286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:15.739259    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerStarted","Data":"b77bc8a749bebe02e4b497436b235baa3c64b4b9b6c29fa364b7c3aa78745b7c"}
Oct 30 21:30:15.741180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:15.741160    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerStarted","Data":"354cf9eac33580239c4e926dea05aba5c5329abd54464ccd0cc738f428af4551"}
Oct 30 21:30:16.218628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.208629    6695 scope.go:122] "RemoveContainer" containerID="389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae"
Oct 30 21:30:16.376888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.376789    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"]
Oct 30 21:30:16.377351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.377067    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy" containerID="cri-o://ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940" gracePeriod=30
Oct 30 21:30:16.757099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.757065    6695 generic.go:356] "Generic (PLEG): container finished" podID="71b8c099-0e21-4843-ac7a-88472df33b27" containerID="b77bc8a749bebe02e4b497436b235baa3c64b4b9b6c29fa364b7c3aa78745b7c" exitCode=60
Oct 30 21:30:16.757321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.757214    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerDied","Data":"b77bc8a749bebe02e4b497436b235baa3c64b4b9b6c29fa364b7c3aa78745b7c"}
Oct 30 21:30:16.757321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.757305    6695 scope.go:122] "RemoveContainer" containerID="72cc0003f09e13860907f3b725dcea539889108b707cd1e0b3557824a7438227"
Oct 30 21:30:16.758001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.757976    6695 scope.go:122] "RemoveContainer" containerID="b77bc8a749bebe02e4b497436b235baa3c64b4b9b6c29fa364b7c3aa78745b7c"
Oct 30 21:30:16.758470 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:16.758444    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5_odh-ai-gateway-infra(71b8c099-0e21-4843-ac7a-88472df33b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" podUID="71b8c099-0e21-4843-ac7a-88472df33b27"
Oct 30 21:30:16.760242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.760219    6695 generic.go:356] "Generic (PLEG): container finished" podID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerID="354cf9eac33580239c4e926dea05aba5c5329abd54464ccd0cc738f428af4551" exitCode=60
Oct 30 21:30:16.760343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.760274    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerDied","Data":"354cf9eac33580239c4e926dea05aba5c5329abd54464ccd0cc738f428af4551"}
Oct 30 21:30:16.760966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.760926    6695 scope.go:122] "RemoveContainer" containerID="354cf9eac33580239c4e926dea05aba5c5329abd54464ccd0cc738f428af4551"
Oct 30 21:30:16.761451 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:16.761393    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-v7rqr_odh-ai-gateway-infra(cd24389a-2e3a-4de2-a26a-6bc05c07e1ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce"
Oct 30 21:30:16.763377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.763349    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerStarted","Data":"7b3ea518bdadaf8370608df1992b9ded434fcb7f94dcf07b65be3e37be797b5c"}
Oct 30 21:30:16.768071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.768046    6695 generic.go:356] "Generic (PLEG): container finished" podID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerID="e57c0d1814d1f1af8a440d828fd83af4a98643c56c462425b05ceb120bbe348b" exitCode=60
Oct 30 21:30:16.768182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.768082    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerDied","Data":"e57c0d1814d1f1af8a440d828fd83af4a98643c56c462425b05ceb120bbe348b"}
Oct 30 21:30:16.768515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.768493    6695 scope.go:122] "RemoveContainer" containerID="e57c0d1814d1f1af8a440d828fd83af4a98643c56c462425b05ceb120bbe348b"
Oct 30 21:30:16.768955 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:16.768913    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4_odh-ai-gateway-infra(0b7d0788-7bd4-4166-9a38-2348592b9cbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc"
Oct 30 21:30:16.775210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.775193    6695 scope.go:122] "RemoveContainer" containerID="0bf550d22f3ea28fb587044ae0a831bf4921812e334dfc7c792aa80ec4481d17"
Oct 30 21:30:16.787425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.787400    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"]
Oct 30 21:30:16.789677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.789595    6695 scope.go:122] "RemoveContainer" containerID="fc0c916ff73cd442888b72576dac80d0ad8ab9afcc9a8eed733025a7edddea63"
Oct 30 21:30:16.826110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.826078    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"]
Oct 30 21:30:16.865658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.865317    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"]
Oct 30 21:30:16.982256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.982171    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"]
Oct 30 21:30:16.982849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.982834    6695 cpu_manager.go:414] "containerMap: removing container" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" containerName="cleanup"
Oct 30 21:30:16.982905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.982850    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" containerName="cleanup"
Oct 30 21:30:16.982905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.982868    6695 cpu_manager.go:414] "containerMap: removing container" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy"
Oct 30 21:30:16.982905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.982873    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy"
Oct 30 21:30:16.983010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.983001    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" containerName="cleanup"
Oct 30 21:30:16.983046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.983017    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy"
Oct 30 21:30:16.988004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:16.987983    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.008011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.007987    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"]
Oct 30 21:30:17.039081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.039048    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:17.133386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.133348    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.133590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.133556    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.133763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.133716    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86d2k\" (UniqueName: \"kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.218640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.218591    6695 scope.go:122] "RemoveContainer" containerID="00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a"
Oct 30 21:30:17.234918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.234889    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d2k\" (UniqueName: \"kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.235077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.235024    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.235120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.235099    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.235771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.235752    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.249860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.249831    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:17.253913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.253889    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86d2k\" (UniqueName: \"kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.257568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.257544    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.301794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.301767    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:17.425452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.425423    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:30:17.450841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.450291    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:30:17.632073 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:30:17.632038    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d8bdde_4f48_4479_ade4_961b6be7001a.slice/crio-e8b5f5da734edf4140d8f75d02dac08007ffa0fa5279dd08848c6347405229f1 WatchSource:0}: Error finding container e8b5f5da734edf4140d8f75d02dac08007ffa0fa5279dd08848c6347405229f1: Status 404 returned error can't find the container with id e8b5f5da734edf4140d8f75d02dac08007ffa0fa5279dd08848c6347405229f1
Oct 30 21:30:17.632494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.632468    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"]
Oct 30 21:30:17.927258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.918242    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerStarted","Data":"d43f59059743837275c74aafe239250eaf6ccd0727551d9386835d0c83f64577"}
Oct 30 21:30:17.940791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.938778    6695 generic.go:356] "Generic (PLEG): container finished" podID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerID="7b3ea518bdadaf8370608df1992b9ded434fcb7f94dcf07b65be3e37be797b5c" exitCode=60
Oct 30 21:30:17.940791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.938974    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerDied","Data":"7b3ea518bdadaf8370608df1992b9ded434fcb7f94dcf07b65be3e37be797b5c"}
Oct 30 21:30:17.940791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.939035    6695 scope.go:122] "RemoveContainer" containerID="389772e8b45a7561affbd5169cffd3eb7c1e5343bb96d7db8346749f0e3d7cae"
Oct 30 21:30:17.940791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.939547    6695 scope.go:122] "RemoveContainer" containerID="7b3ea518bdadaf8370608df1992b9ded434fcb7f94dcf07b65be3e37be797b5c"
Oct 30 21:30:17.962627 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:17.944292    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j_odh-ai-gateway-infra(630efc9b-7c36-4514-b4f1-4a6c1643a7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0"
Oct 30 21:30:17.962627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.958405    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx" event={"ID":"c2d8bdde-4f48-4479-ade4-961b6be7001a","Type":"ContainerStarted","Data":"e8b5f5da734edf4140d8f75d02dac08007ffa0fa5279dd08848c6347405229f1"}
Oct 30 21:30:17.996536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:17.996508    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"]
Oct 30 21:30:18.031265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.031236    6695 scope.go:122] "RemoveContainer" containerID="9f5f5ac01e4ba1ff8bc536cf1e1c809f76cfddd2e7db92cb94e9f1f4b86a5813"
Oct 30 21:30:18.184672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.184640    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:18.208619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.208288    6695 scope.go:122] "RemoveContainer" containerID="c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a"
Oct 30 21:30:18.271742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.271701    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4\" (UniqueName: \"kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4\") pod \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") "
Oct 30 21:30:18.271944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.271804    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca\") pod \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\" (UID: \"0b7d0788-7bd4-4166-9a38-2348592b9cbc\") "
Oct 30 21:30:18.272252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.272221    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca" pod "0b7d0788-7bd4-4166-9a38-2348592b9cbc" (UID: "0b7d0788-7bd4-4166-9a38-2348592b9cbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.272655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.272584    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7d0788-7bd4-4166-9a38-2348592b9cbc-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.274138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.274104    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4" pod "0b7d0788-7bd4-4166-9a38-2348592b9cbc" (UID: "0b7d0788-7bd4-4166-9a38-2348592b9cbc"). InnerVolumeSpecName "kube-api-access-xmtc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.286450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.286426    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:18.298163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.298139    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:18.374725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.373273    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m\" (UniqueName: \"kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m\") pod \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") "
Oct 30 21:30:18.381419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.379177    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m" pod "cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" (UID: "cd24389a-2e3a-4de2-a26a-6bc05c07e1ce"). InnerVolumeSpecName "kube-api-access-svn4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.381642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.381506    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca\") pod \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\" (UID: \"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce\") "
Oct 30 21:30:18.382109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.382070    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca" pod "cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" (UID: "cd24389a-2e3a-4de2-a26a-6bc05c07e1ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.383046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.383019    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svn4m\" (UniqueName: \"kubernetes.io/projected/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-kube-api-access-svn4m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.383046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.383043    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.383211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.383056    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtc4\" (UniqueName: \"kubernetes.io/projected/0b7d0788-7bd4-4166-9a38-2348592b9cbc-kube-api-access-xmtc4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.489821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.489272    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw\" (UniqueName: \"kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw\") pod \"71b8c099-0e21-4843-ac7a-88472df33b27\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") "
Oct 30 21:30:18.489821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.489515    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca\") pod \"71b8c099-0e21-4843-ac7a-88472df33b27\" (UID: \"71b8c099-0e21-4843-ac7a-88472df33b27\") "
Oct 30 21:30:18.552683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.550848    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca" pod "71b8c099-0e21-4843-ac7a-88472df33b27" (UID: "71b8c099-0e21-4843-ac7a-88472df33b27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:18.603687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.601547    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b8c099-0e21-4843-ac7a-88472df33b27-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.611625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.611121    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw" pod "71b8c099-0e21-4843-ac7a-88472df33b27" (UID: "71b8c099-0e21-4843-ac7a-88472df33b27"). InnerVolumeSpecName "kube-api-access-nvtnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:18.629679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.629420    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"]
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631671    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631700    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631722    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631729    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631741    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631804    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631820    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631827    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631842    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631849    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631882    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631888    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631924    6695 cpu_manager.go:414] "containerMap: removing container" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631931    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631957    6695 cpu_manager.go:414] "containerMap: removing container" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631964    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631991    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.631999    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632161    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="251d9852-8145-4cc5-a9ea-7e295dc460c0" containerName="istio-proxy"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632181    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632191    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632199    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="689264b2-c504-4124-b951-1c6ef3f05caa" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632211    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632220    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632228    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:18.644666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.632238    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:18.669651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.652543    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.706627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.705642    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"]
Oct 30 21:30:18.724971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.714310    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtnw\" (UniqueName: \"kubernetes.io/projected/71b8c099-0e21-4843-ac7a-88472df33b27-kube-api-access-nvtnw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:18.815775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.815734    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.815938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.815885    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7j67\" (UniqueName: \"kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.816159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.816130    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.917738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.917661    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.917888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.917803    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7j67\" (UniqueName: \"kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.917992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.917975    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.918810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.918784    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.920208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.920183    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.940138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.940106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7j67\" (UniqueName: \"kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67\") pod \"maas-api-revoke-sub-f6cd92d4-962mb\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:18.969693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.969643    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4" event={"ID":"0b7d0788-7bd4-4166-9a38-2348592b9cbc","Type":"ContainerDied","Data":"f9bca7545fe3dc3ec4ad88b4563e81ab70724150048b473cb0135f820edbdafd"}
Oct 30 21:30:18.969864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.969723    6695 scope.go:122] "RemoveContainer" containerID="e57c0d1814d1f1af8a440d828fd83af4a98643c56c462425b05ceb120bbe348b"
Oct 30 21:30:18.969864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.969654    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"
Oct 30 21:30:18.971526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.971505    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"
Oct 30 21:30:18.972227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.972205    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5" event={"ID":"71b8c099-0e21-4843-ac7a-88472df33b27","Type":"ContainerDied","Data":"457b26ef0e542248418e2535142d0b649c4e8071a030885f4d1eff314f071f8f"}
Oct 30 21:30:18.975326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.975298    6695 generic.go:356] "Generic (PLEG): container finished" podID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerID="d43f59059743837275c74aafe239250eaf6ccd0727551d9386835d0c83f64577" exitCode=60
Oct 30 21:30:18.975451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.975372    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerDied","Data":"d43f59059743837275c74aafe239250eaf6ccd0727551d9386835d0c83f64577"}
Oct 30 21:30:18.975851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.975831    6695 scope.go:122] "RemoveContainer" containerID="d43f59059743837275c74aafe239250eaf6ccd0727551d9386835d0c83f64577"
Oct 30 21:30:18.976215 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:18.976192    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9_odh-ai-gateway-infra(c71aaa5e-9194-4ba2-b3db-fd0f23cbea14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14"
Oct 30 21:30:18.978396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.978374    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"
Oct 30 21:30:18.978497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.978432    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr" event={"ID":"cd24389a-2e3a-4de2-a26a-6bc05c07e1ce","Type":"ContainerDied","Data":"9a8f229f2e84bb88ab8c0dab8a7b4528c2551f4de72990602bb505fb1f70e223"}
Oct 30 21:30:18.983617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.983585    6695 scope.go:122] "RemoveContainer" containerID="b77bc8a749bebe02e4b497436b235baa3c64b4b9b6c29fa364b7c3aa78745b7c"
Oct 30 21:30:18.986349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.986317    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerStarted","Data":"be16b26efbb4e8501846404305c7274cec25bc3c53fe7a69b654d8ac241cfa3f"}
Oct 30 21:30:18.988452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.988425    6695 generic.go:356] "Generic (PLEG): container finished" podID="c2d8bdde-4f48-4479-ade4-961b6be7001a" containerID="0015ac98f5ceeecdfdbec5b362e65a1e8f29f3c83118d158eb23d2dd31a07383" exitCode=0
Oct 30 21:30:18.988542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.988488    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx" event={"ID":"c2d8bdde-4f48-4479-ade4-961b6be7001a","Type":"ContainerDied","Data":"0015ac98f5ceeecdfdbec5b362e65a1e8f29f3c83118d158eb23d2dd31a07383"}
Oct 30 21:30:18.998176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:18.998154    6695 scope.go:122] "RemoveContainer" containerID="00e32589b85355107c8116b13d238cc7f4f74624f045b002d443b8e96aaa352a"
Oct 30 21:30:19.015467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.015438    6695 scope.go:122] "RemoveContainer" containerID="354cf9eac33580239c4e926dea05aba5c5329abd54464ccd0cc738f428af4551"
Oct 30 21:30:19.033411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.033378    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"]
Oct 30 21:30:19.042248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.041960    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"]
Oct 30 21:30:19.042248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.042042    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:19.052855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.052813    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889930-5f8p4"]
Oct 30 21:30:19.320716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.305844    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:19.345662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.335168    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m\" (UniqueName: \"kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m\") pod \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") "
Oct 30 21:30:19.345662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.335342    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca\") pod \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\" (UID: \"630efc9b-7c36-4514-b4f1-4a6c1643a7d0\") "
Oct 30 21:30:19.345662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.337873    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca" pod "630efc9b-7c36-4514-b4f1-4a6c1643a7d0" (UID: "630efc9b-7c36-4514-b4f1-4a6c1643a7d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:19.357634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.357247    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m" pod "630efc9b-7c36-4514-b4f1-4a6c1643a7d0" (UID: "630efc9b-7c36-4514-b4f1-4a6c1643a7d0"). InnerVolumeSpecName "kube-api-access-k2p4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:19.425965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.420489    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" path="/var/lib/kubelet/pods/0b7d0788-7bd4-4166-9a38-2348592b9cbc/volumes"
Oct 30 21:30:19.460387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.455957    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:19.460387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.455993    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2p4m\" (UniqueName: \"kubernetes.io/projected/630efc9b-7c36-4514-b4f1-4a6c1643a7d0-kube-api-access-k2p4m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:19.489328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.489241    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"]
Oct 30 21:30:19.689756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.689620    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c6dab8-29889930-zhnv5"]
Oct 30 21:30:19.782228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.782133    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"]
Oct 30 21:30:19.789902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.789821    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-v7rqr"]
Oct 30 21:30:19.813314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:19.813284    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"]
Oct 30 21:30:20.000406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.000304    6695 generic.go:356] "Generic (PLEG): container finished" podID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerID="be16b26efbb4e8501846404305c7274cec25bc3c53fe7a69b654d8ac241cfa3f" exitCode=60
Oct 30 21:30:20.000406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.000387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerDied","Data":"be16b26efbb4e8501846404305c7274cec25bc3c53fe7a69b654d8ac241cfa3f"}
Oct 30 21:30:20.000708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.000451    6695 scope.go:122] "RemoveContainer" containerID="c7dd2aed29423dd180a0959ffd3838f147b275227b82c17369512b41e54ece5a"
Oct 30 21:30:20.000987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.000965    6695 scope.go:122] "RemoveContainer" containerID="be16b26efbb4e8501846404305c7274cec25bc3c53fe7a69b654d8ac241cfa3f"
Oct 30 21:30:20.001418 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:20.001386    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x_odh-ai-gateway-infra(b6db89eb-7b62-4044-808a-8f64364c5eed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed"
Oct 30 21:30:20.024854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.024810    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb" event={"ID":"8e9d02c2-695a-418f-8c9b-79412b38c072","Type":"ContainerStarted","Data":"119ea9a3d3efbd00cb501cca8f1340cceb39ba16acbc5ca275f852ee16930cae"}
Oct 30 21:30:20.036435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.034186    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"
Oct 30 21:30:20.043998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.043956    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j" event={"ID":"630efc9b-7c36-4514-b4f1-4a6c1643a7d0","Type":"ContainerDied","Data":"b92d72ff2942aa1ce8479eb0886a2add0410139e6e69927e16c7cfa726ff5e6b"}
Oct 30 21:30:20.046463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.045861    6695 scope.go:122] "RemoveContainer" containerID="7b3ea518bdadaf8370608df1992b9ded434fcb7f94dcf07b65be3e37be797b5c"
Oct 30 21:30:20.070885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.070856    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"]
Oct 30 21:30:20.077224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.077173    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df70818f-l82wn"]
Oct 30 21:30:20.157397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.157332    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"]
Oct 30 21:30:20.161462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.161432    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889930-v7h9j"]
Oct 30 21:30:20.282429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.282376    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"]
Oct 30 21:30:20.349797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.349756    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:20.379231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.379189    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca\") pod \"c2d8bdde-4f48-4479-ade4-961b6be7001a\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") "
Oct 30 21:30:20.379374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.379266    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca\") pod \"c2d8bdde-4f48-4479-ade4-961b6be7001a\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") "
Oct 30 21:30:20.379374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.379297    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k\" (UniqueName: \"kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k\") pod \"c2d8bdde-4f48-4479-ade4-961b6be7001a\" (UID: \"c2d8bdde-4f48-4479-ade4-961b6be7001a\") "
Oct 30 21:30:20.379716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.379690    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca" pod "c2d8bdde-4f48-4479-ade4-961b6be7001a" (UID: "c2d8bdde-4f48-4479-ade4-961b6be7001a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.380102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.380083    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.381505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.381479    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k" pod "c2d8bdde-4f48-4479-ade4-961b6be7001a" (UID: "c2d8bdde-4f48-4479-ade4-961b6be7001a"). InnerVolumeSpecName "kube-api-access-86d2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.381741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.381717    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca" pod "c2d8bdde-4f48-4479-ade4-961b6be7001a" (UID: "c2d8bdde-4f48-4479-ade4-961b6be7001a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:20.402267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.402239    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:20.491428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.482103    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q\" (UniqueName: \"kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q\") pod \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") "
Oct 30 21:30:20.491428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.489968    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca\") pod \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\" (UID: \"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14\") "
Oct 30 21:30:20.491428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.490939    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d8bdde-4f48-4479-ade4-961b6be7001a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.491428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.490964    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86d2k\" (UniqueName: \"kubernetes.io/projected/c2d8bdde-4f48-4479-ade4-961b6be7001a-kube-api-access-86d2k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.491428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.491340    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca" pod "c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" (UID: "c71aaa5e-9194-4ba2-b3db-fd0f23cbea14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.502010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.501969    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q" pod "c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" (UID: "c71aaa5e-9194-4ba2-b3db-fd0f23cbea14"). InnerVolumeSpecName "kube-api-access-cgr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:20.637803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.627043    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr7q\" (UniqueName: \"kubernetes.io/projected/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-kube-api-access-cgr7q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:20.637803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:20.627086    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.051510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.051479    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"
Oct 30 21:30:21.051985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.051479    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9" event={"ID":"c71aaa5e-9194-4ba2-b3db-fd0f23cbea14","Type":"ContainerDied","Data":"618a381a77e827db9b8c6a2682b9475615ee4be3b9583342fefd924512d9789c"}
Oct 30 21:30:21.051985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.051686    6695 scope.go:122] "RemoveContainer" containerID="d43f59059743837275c74aafe239250eaf6ccd0727551d9386835d0c83f64577"
Oct 30 21:30:21.053574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.053535    6695 generic.go:356] "Generic (PLEG): container finished" podID="8e9d02c2-695a-418f-8c9b-79412b38c072" containerID="0ae9a856f9c501d5cb60b856fbfb4893dfd54ce579e57a4ee350b993159a7ee0" exitCode=0
Oct 30 21:30:21.053729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.053593    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb" event={"ID":"8e9d02c2-695a-418f-8c9b-79412b38c072","Type":"ContainerDied","Data":"0ae9a856f9c501d5cb60b856fbfb4893dfd54ce579e57a4ee350b993159a7ee0"}
Oct 30 21:30:21.065765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.065722    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx" event={"ID":"c2d8bdde-4f48-4479-ade4-961b6be7001a","Type":"ContainerDied","Data":"e8b5f5da734edf4140d8f75d02dac08007ffa0fa5279dd08848c6347405229f1"}
Oct 30 21:30:21.065893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.065790    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b5f5da734edf4140d8f75d02dac08007ffa0fa5279dd08848c6347405229f1"
Oct 30 21:30:21.066320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.066296    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"
Oct 30 21:30:21.190218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.190123    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"]
Oct 30 21:30:21.203491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.203455    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889930-qnrj9"]
Oct 30 21:30:21.250152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.249893    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" path="/var/lib/kubelet/pods/630efc9b-7c36-4514-b4f1-4a6c1643a7d0/volumes"
Oct 30 21:30:21.250524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.250501    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" path="/var/lib/kubelet/pods/71b8c099-0e21-4843-ac7a-88472df33b27/volumes"
Oct 30 21:30:21.251044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.251026    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f40ab1-058d-49d6-8098-6b92e5421aa2" path="/var/lib/kubelet/pods/74f40ab1-058d-49d6-8098-6b92e5421aa2/volumes"
Oct 30 21:30:21.251802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.251781    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" path="/var/lib/kubelet/pods/c71aaa5e-9194-4ba2-b3db-fd0f23cbea14/volumes"
Oct 30 21:30:21.252235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.252218    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" path="/var/lib/kubelet/pods/cd24389a-2e3a-4de2-a26a-6bc05c07e1ce/volumes"
Oct 30 21:30:21.260672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.260653    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:21.362684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.362576    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca" pod "b6db89eb-7b62-4044-808a-8f64364c5eed" (UID: "b6db89eb-7b62-4044-808a-8f64364c5eed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:21.365491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.365464    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca\") pod \"b6db89eb-7b62-4044-808a-8f64364c5eed\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") "
Oct 30 21:30:21.365717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.365527    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k\" (UniqueName: \"kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k\") pod \"b6db89eb-7b62-4044-808a-8f64364c5eed\" (UID: \"b6db89eb-7b62-4044-808a-8f64364c5eed\") "
Oct 30 21:30:21.366355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.366320    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6db89eb-7b62-4044-808a-8f64364c5eed-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.376487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.376414    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k" pod "b6db89eb-7b62-4044-808a-8f64364c5eed" (UID: "b6db89eb-7b62-4044-808a-8f64364c5eed"). InnerVolumeSpecName "kube-api-access-9452k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:21.468008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.467919    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9452k\" (UniqueName: \"kubernetes.io/projected/b6db89eb-7b62-4044-808a-8f64364c5eed-kube-api-access-9452k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.746115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.746040    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:30:21.876795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.876759    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.876832    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.876866    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.876963    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877237    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877306    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877357    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877392    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877471    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy\") pod \"ef73eff5-5da0-41af-86f6-744c054a8ef8\" (UID: \"ef73eff5-5da0-41af-86f6-744c054a8ef8\") "
Oct 30 21:30:21.877763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877719    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:21.878020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.877991    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:21.878091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878066    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:21.878145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878119    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:21.878281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878266    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.878348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878285    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef73eff5-5da0-41af-86f6-744c054a8ef8-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.878348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878296    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.878348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878304    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.878815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.878783    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:21.879889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.879865    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:21.880444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.880421    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:21.881476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.881341    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:21.881808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.881767    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t" pod "ef73eff5-5da0-41af-86f6-744c054a8ef8" (UID: "ef73eff5-5da0-41af-86f6-744c054a8ef8"). InnerVolumeSpecName "kube-api-access-6l78t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:21.979716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.979682    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.979716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.979714    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.979932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.979729    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l78t\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-kube-api-access-6l78t\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.979932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.979742    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:21.979932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:21.979756    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef73eff5-5da0-41af-86f6-744c054a8ef8-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:22.076926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.076892    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"
Oct 30 21:30:22.077395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.076904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x" event={"ID":"b6db89eb-7b62-4044-808a-8f64364c5eed","Type":"ContainerDied","Data":"44632796795d1fa3f27e56446d66fc729a635bc0f5103549aae2dcc97e1b1183"}
Oct 30 21:30:22.077535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.077430    6695 scope.go:122] "RemoveContainer" containerID="be16b26efbb4e8501846404305c7274cec25bc3c53fe7a69b654d8ac241cfa3f"
Oct 30 21:30:22.079311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.079283    6695 generic.go:356] "Generic (PLEG): container finished" podID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerID="ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940" exitCode=0
Oct 30 21:30:22.079438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.079404    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" event={"ID":"ef73eff5-5da0-41af-86f6-744c054a8ef8","Type":"ContainerDied","Data":"ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940"}
Oct 30 21:30:22.079512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.079496    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48" event={"ID":"ef73eff5-5da0-41af-86f6-744c054a8ef8","Type":"ContainerDied","Data":"afa2fc785392e347adcce7fec896fb8859f035a7eeea504e8a2e19a5db8e5b07"}
Oct 30 21:30:22.079642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.079624    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"
Oct 30 21:30:22.090773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.090749    6695 scope.go:122] "RemoveContainer" containerID="ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940"
Oct 30 21:30:22.105138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.105094    6695 scope.go:122] "RemoveContainer" containerID="1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"
Oct 30 21:30:22.148665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.146163    6695 scope.go:122] "RemoveContainer" containerID="ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940"
Oct 30 21:30:22.148665 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:22.146650    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940\": container with ID starting with ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940 not found: ID does not exist" containerID="ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940"
Oct 30 21:30:22.148665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.146695    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940"} err="failed to get container status \"ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940\": rpc error: code = NotFound desc = could not find container \"ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940\": container with ID starting with ed87c8398c41fb13ae0c58e66d4b0da57924be8676c196a2a3db8e89e602a940 not found: ID does not exist"
Oct 30 21:30:22.148665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.146724    6695 scope.go:122] "RemoveContainer" containerID="1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"
Oct 30 21:30:22.148665 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:22.147042    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784\": container with ID starting with 1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784 not found: ID does not exist" containerID="1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"
Oct 30 21:30:22.148665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.147067    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784"} err="failed to get container status \"1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784\": rpc error: code = NotFound desc = could not find container \"1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784\": container with ID starting with 1cb6231c4d42a149459be58a0d82f1cf285b2702ecf8cf36aac80cd6ef689784 not found: ID does not exist"
Oct 30 21:30:22.164951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.164907    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"]
Oct 30 21:30:22.169965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.169915    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b13793-openshift-default-787f4cc8f5-zvf48"]
Oct 30 21:30:22.202937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.201743    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"]
Oct 30 21:30:22.215627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.210288    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ab5774-29889930-lzv9x"]
Oct 30 21:30:22.304924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.304892    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:22.388801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.388770    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca\") pod \"8e9d02c2-695a-418f-8c9b-79412b38c072\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") "
Oct 30 21:30:22.388996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.388851    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca\") pod \"8e9d02c2-695a-418f-8c9b-79412b38c072\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") "
Oct 30 21:30:22.388996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.388884    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67\" (UniqueName: \"kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67\") pod \"8e9d02c2-695a-418f-8c9b-79412b38c072\" (UID: \"8e9d02c2-695a-418f-8c9b-79412b38c072\") "
Oct 30 21:30:22.389189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.389160    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca" pod "8e9d02c2-695a-418f-8c9b-79412b38c072" (UID: "8e9d02c2-695a-418f-8c9b-79412b38c072"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.389517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.389494    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:22.391253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.391230    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67" pod "8e9d02c2-695a-418f-8c9b-79412b38c072" (UID: "8e9d02c2-695a-418f-8c9b-79412b38c072"). InnerVolumeSpecName "kube-api-access-w7j67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.391804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.391787    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca" pod "8e9d02c2-695a-418f-8c9b-79412b38c072" (UID: "8e9d02c2-695a-418f-8c9b-79412b38c072"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:22.499627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.492166    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e9d02c2-695a-418f-8c9b-79412b38c072-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:22.499627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:22.492209    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7j67\" (UniqueName: \"kubernetes.io/projected/8e9d02c2-695a-418f-8c9b-79412b38c072-kube-api-access-w7j67\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:23.090235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:23.090195    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb" event={"ID":"8e9d02c2-695a-418f-8c9b-79412b38c072","Type":"ContainerDied","Data":"119ea9a3d3efbd00cb501cca8f1340cceb39ba16acbc5ca275f852ee16930cae"}
Oct 30 21:30:23.091135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:23.090264    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119ea9a3d3efbd00cb501cca8f1340cceb39ba16acbc5ca275f852ee16930cae"
Oct 30 21:30:23.091135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:23.090345    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"
Oct 30 21:30:23.233628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:23.230503    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" path="/var/lib/kubelet/pods/b6db89eb-7b62-4044-808a-8f64364c5eed/volumes"
Oct 30 21:30:23.233628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:23.231373    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" path="/var/lib/kubelet/pods/ef73eff5-5da0-41af-86f6-744c054a8ef8/volumes"
Oct 30 21:30:25.508230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.508188    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7400-c0203a"]
Oct 30 21:30:25.509221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509195    6695 cpu_manager.go:414] "containerMap: removing container" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:25.509269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509226    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:25.509269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509248    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy"
Oct 30 21:30:25.509269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509256    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509270    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509279    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509293    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509300    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509321    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c2d8bdde-4f48-4479-ade4-961b6be7001a" containerName="revoke-keys"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509330    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d8bdde-4f48-4479-ade4-961b6be7001a" containerName="revoke-keys"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509347    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509354    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509363    6695 cpu_manager.go:414] "containerMap: removing container" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:25.509371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509370    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509398    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509404    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509420    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8e9d02c2-695a-418f-8c9b-79412b38c072" containerName="revoke-keys"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509427    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9d02c2-695a-418f-8c9b-79412b38c072" containerName="revoke-keys"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509455    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509462    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509476    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509483    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509650    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509665    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509676    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:25.509690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509686    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d8bdde-4f48-4479-ade4-961b6be7001a" containerName="revoke-keys"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509697    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509706    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509718    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509728    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd24389a-2e3a-4de2-a26a-6bc05c07e1ce" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509735    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9d02c2-695a-418f-8c9b-79412b38c072" containerName="revoke-keys"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509747    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509756    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b8c099-0e21-4843-ac7a-88472df33b27" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509764    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7d0788-7bd4-4166-9a38-2348592b9cbc" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509771    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:25.510030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.509788    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy"
Oct 30 21:30:25.543879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.543833    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7400-c0203a"]
Oct 30 21:30:25.544064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.543950    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:25.559119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.559087    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vfm5w\""
Oct 30 21:30:25.573681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.572630    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxplv\" (UniqueName: \"kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv\") pod \"test-curl-7400-c0203a\" (UID: \"702a5ec8-413e-4985-8a49-e982c1a8ec8b\") " pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:25.674837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.674796    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxplv\" (UniqueName: \"kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv\") pod \"test-curl-7400-c0203a\" (UID: \"702a5ec8-413e-4985-8a49-e982c1a8ec8b\") " pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:25.697187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.697157    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxplv\" (UniqueName: \"kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv\") pod \"test-curl-7400-c0203a\" (UID: \"702a5ec8-413e-4985-8a49-e982c1a8ec8b\") " pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:25.870652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:25.862879    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:26.054106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:26.054021    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"]
Oct 30 21:30:26.069798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:26.069740    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89153b7e-q6c56"]
Oct 30 21:30:26.400314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:26.400214    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7400-c0203a"]
Oct 30 21:30:26.407896 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:30:26.407858    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod702a5ec8_413e_4985_8a49_e982c1a8ec8b.slice/crio-2a2918a2ebd8a607a8b186fe74fa592d5dc7fca1988398ab65f1f9a44dcc7510 WatchSource:0}: Error finding container 2a2918a2ebd8a607a8b186fe74fa592d5dc7fca1988398ab65f1f9a44dcc7510: Status 404 returned error can't find the container with id 2a2918a2ebd8a607a8b186fe74fa592d5dc7fca1988398ab65f1f9a44dcc7510
Oct 30 21:30:27.167623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:27.163128    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7400-c0203a" event={"ID":"702a5ec8-413e-4985-8a49-e982c1a8ec8b","Type":"ContainerStarted","Data":"ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018"}
Oct 30 21:30:27.167623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:27.163183    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7400-c0203a" event={"ID":"702a5ec8-413e-4985-8a49-e982c1a8ec8b","Type":"ContainerStarted","Data":"2a2918a2ebd8a607a8b186fe74fa592d5dc7fca1988398ab65f1f9a44dcc7510"}
Oct 30 21:30:27.207107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:27.207028    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7400-c0203a" podStartSLOduration=2.207005061 podStartE2EDuration="2.207005061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:27.199534467 +0000 UTC m=+1210.121992766" watchObservedRunningTime="2026-10-30 21:30:27.207005061 +0000 UTC m=+1210.129463359"
Oct 30 21:30:27.227674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:27.227638    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2bae61-33c4-4d29-80b9-0fab06281e84" path="/var/lib/kubelet/pods/7d2bae61-33c4-4d29-80b9-0fab06281e84/volumes"
Oct 30 21:30:27.292522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:27.292480    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"]
Oct 30 21:30:27.324845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:27.324791    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"]
Oct 30 21:30:28.184629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.148647    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7400-c0203a"]
Oct 30 21:30:28.269685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.265087    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7400-c0203a"]
Oct 30 21:30:28.415020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.414989    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"]
Oct 30 21:30:28.415656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415641    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415657    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef73eff5-5da0-41af-86f6-744c054a8ef8" containerName="istio-proxy"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415669    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415675    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415692    6695 cpu_manager.go:414] "containerMap: removing container" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415696    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415705    6695 cpu_manager.go:414] "containerMap: removing container" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" containerName="test-curl-7400-c0203a"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415709    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" containerName="test-curl-7400-c0203a"
Oct 30 21:30:28.415724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415728    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:28.415970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415732    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6db89eb-7b62-4044-808a-8f64364c5eed" containerName="cleanup"
Oct 30 21:30:28.415970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415834    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71aaa5e-9194-4ba2-b3db-fd0f23cbea14" containerName="cleanup"
Oct 30 21:30:28.415970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415844    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="630efc9b-7c36-4514-b4f1-4a6c1643a7d0" containerName="cleanup"
Oct 30 21:30:28.415970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.415850    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" containerName="test-curl-7400-c0203a"
Oct 30 21:30:28.420530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.420513    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.438573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.438495    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"]
Oct 30 21:30:28.445459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.445344    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:28.445459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.445425    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:28.445840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.445807    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:30:28.446027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.446004    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:28.559623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.558733    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.559623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.558793    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.559623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.558899    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdm7w\" (UniqueName: \"kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.660851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.660805    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.661014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.660921    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm7w\" (UniqueName: \"kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.661199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.661177    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.662002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.661979    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.663212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.663192    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.679533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.679504    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm7w\" (UniqueName: \"kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w\") pod \"maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:28.737755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:28.737674    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:29.052056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.052026    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"]
Oct 30 21:30:29.055042 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:30:29.055014    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526e7c21_fdfd_43b2_aef5_ec7d1c1b73df.slice/crio-70d8d4023a10ab6785ee86fe0279c516cc45189ec5e8f570a9fb21486fadcb0e WatchSource:0}: Error finding container 70d8d4023a10ab6785ee86fe0279c516cc45189ec5e8f570a9fb21486fadcb0e: Status 404 returned error can't find the container with id 70d8d4023a10ab6785ee86fe0279c516cc45189ec5e8f570a9fb21486fadcb0e
Oct 30 21:30:29.274090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.274051    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g" event={"ID":"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df","Type":"ContainerStarted","Data":"70d8d4023a10ab6785ee86fe0279c516cc45189ec5e8f570a9fb21486fadcb0e"}
Oct 30 21:30:29.274497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.274291    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7400-c0203a" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" containerName="test-curl-7400-c0203a" containerID="cri-o://ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018" gracePeriod=2
Oct 30 21:30:29.718317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.718233    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"]
Oct 30 21:30:29.724073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.724044    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.742357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.742324    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"]
Oct 30 21:30:29.887936 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.887887    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.888263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.888226    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jtb\" (UniqueName: \"kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.888424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.888408    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.996444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.990749    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.996444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.990850    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jtb\" (UniqueName: \"kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.996444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.990907    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:29.998164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.997363    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:30.002300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:29.999905    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:30.020689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.020653    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jtb\" (UniqueName: \"kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb\") pod \"maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:30.045839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.045812    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:30.283832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.283800    6695 generic.go:356] "Generic (PLEG): container finished" podID="526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" containerID="b9a17974b956ee90c33fba4e95bb49d7ed1ba65ee6390488062748fd82d62e73" exitCode=0
Oct 30 21:30:30.284318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.283845    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g" event={"ID":"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df","Type":"ContainerDied","Data":"b9a17974b956ee90c33fba4e95bb49d7ed1ba65ee6390488062748fd82d62e73"}
Oct 30 21:30:30.285734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.285711    6695 generic.go:356] "Generic (PLEG): container finished" podID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerID="55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc" exitCode=0
Oct 30 21:30:30.285862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.285781    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" event={"ID":"8a4cd3c8-25a0-477a-976c-b591bb067e68","Type":"ContainerDied","Data":"55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"}
Oct 30 21:30:30.286206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.286193    6695 scope.go:122] "RemoveContainer" containerID="55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"
Oct 30 21:30:30.287857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.287813    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:30:30.287971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.287898    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:30:30.287971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.287941    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:30:30.373521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:30.373459    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"]
Oct 30 21:30:31.074104 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:31.074065    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:30:31.142823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.142786    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:30:31.296901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.296868    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j" event={"ID":"be8366fb-2166-4717-9f0b-333d7851b6f7","Type":"ContainerStarted","Data":"a3119f329f01709916975ccd66eb9f48c06f82bac2807620a12e679edde975be"}
Oct 30 21:30:31.297272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.296927    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j" event={"ID":"be8366fb-2166-4717-9f0b-333d7851b6f7","Type":"ContainerStarted","Data":"cca280a6254762247599b268a4538581ea0e8799fb2ff5b57071a1130b26b0b0"}
Oct 30 21:30:31.298864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.298841    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" event={"ID":"8a4cd3c8-25a0-477a-976c-b591bb067e68","Type":"ContainerStarted","Data":"f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489"}
Oct 30 21:30:31.361862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.361811    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j" podStartSLOduration=2.361797456 podStartE2EDuration="2.361797456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:31.354535806 +0000 UTC m=+1214.276994103" watchObservedRunningTime="2026-10-30 21:30:31.361797456 +0000 UTC m=+1214.284255761"
Oct 30 21:30:31.588295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.588273    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:31.597000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.596945    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:31.655885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.655849    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w\" (UniqueName: \"kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w\") pod \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") "
Oct 30 21:30:31.656106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.655920    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv\" (UniqueName: \"kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv\") pod \"702a5ec8-413e-4985-8a49-e982c1a8ec8b\" (UID: \"702a5ec8-413e-4985-8a49-e982c1a8ec8b\") "
Oct 30 21:30:31.656106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.655950    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca\") pod \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") "
Oct 30 21:30:31.656271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.656243    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca\") pod \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\" (UID: \"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df\") "
Oct 30 21:30:31.656920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.656890    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca" pod "526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" (UID: "526e7c21-fdfd-43b2-aef5-ec7d1c1b73df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:31.657095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.657075    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:31.658649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.658594    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca" pod "526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" (UID: "526e7c21-fdfd-43b2-aef5-ec7d1c1b73df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:31.658885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.658856    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w" pod "526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" (UID: "526e7c21-fdfd-43b2-aef5-ec7d1c1b73df"). InnerVolumeSpecName "kube-api-access-bdm7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:31.658981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.658896    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv" pod "702a5ec8-413e-4985-8a49-e982c1a8ec8b" (UID: "702a5ec8-413e-4985-8a49-e982c1a8ec8b"). InnerVolumeSpecName "kube-api-access-kxplv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:31.758591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.758547    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:31.758591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.758592    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdm7w\" (UniqueName: \"kubernetes.io/projected/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df-kube-api-access-bdm7w\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:31.758591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.758624    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxplv\" (UniqueName: \"kubernetes.io/projected/702a5ec8-413e-4985-8a49-e982c1a8ec8b-kube-api-access-kxplv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:31.761684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.761649    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:30:31.762676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.762655    6695 patch_prober.go:29] interesting pod/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:30:31.762750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:31.762701    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 21:30:32.293836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.293796    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm" podUID="f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" containerName="payload-processing" containerID="cri-o://4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988" gracePeriod=25
Oct 30 21:30:32.308742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.308717    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"
Oct 30 21:30:32.308742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.308730    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g" event={"ID":"526e7c21-fdfd-43b2-aef5-ec7d1c1b73df","Type":"ContainerDied","Data":"70d8d4023a10ab6785ee86fe0279c516cc45189ec5e8f570a9fb21486fadcb0e"}
Oct 30 21:30:32.309148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.308783    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d8d4023a10ab6785ee86fe0279c516cc45189ec5e8f570a9fb21486fadcb0e"
Oct 30 21:30:32.310404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.310378    6695 generic.go:356] "Generic (PLEG): container finished" podID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" containerID="ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018" exitCode=137
Oct 30 21:30:32.310526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.310422    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:32.310526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.310475    6695 scope.go:122] "RemoveContainer" containerID="ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018"
Oct 30 21:30:32.326790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.326708    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk" podUID="3173f1d6-9c04-4826-b329-004c73fb2958" containerName="payload-pre-processing" containerID="cri-o://d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065" gracePeriod=25
Oct 30 21:30:32.340627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.339243    6695 generic.go:356] "Generic (PLEG): container finished" podID="be8366fb-2166-4717-9f0b-333d7851b6f7" containerID="a3119f329f01709916975ccd66eb9f48c06f82bac2807620a12e679edde975be" exitCode=0
Oct 30 21:30:32.340627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.339368    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j" event={"ID":"be8366fb-2166-4717-9f0b-333d7851b6f7","Type":"ContainerDied","Data":"a3119f329f01709916975ccd66eb9f48c06f82bac2807620a12e679edde975be"}
Oct 30 21:30:32.455064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.455043    6695 scope.go:122] "RemoveContainer" containerID="ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018"
Oct 30 21:30:32.455434 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:32.455406    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018\": container with ID starting with ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018 not found: ID does not exist" containerID="ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018"
Oct 30 21:30:32.455493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.455442    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018"} err="failed to get container status \"ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018\": rpc error: code = NotFound desc = could not find container \"ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018\": container with ID starting with ad79c281fb0b45700d3a7e0b164ce62287502877d5e7c72744026902ab4c5018 not found: ID does not exist"
Oct 30 21:30:32.481704 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:32.481666    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7400-c0203a\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:32.494635 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:32.494239    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7400-c0203a\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:32.665503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.665471    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:30:32.742617 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:32.742544    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7400-c0203a\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:32.753416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.753386    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:30:32.761693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.761662    6695 patch_prober.go:29] interesting pod/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:30:32.761817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.761713    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 21:30:32.779859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.779832    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume\") pod \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") "
Oct 30 21:30:32.780002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.779901    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq\" (UniqueName: \"kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq\") pod \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\" (UID: \"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da\") "
Oct 30 21:30:32.780101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.780013    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume\") pod \"3173f1d6-9c04-4826-b329-004c73fb2958\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") "
Oct 30 21:30:32.780169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.780105    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl\" (UniqueName: \"kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl\") pod \"3173f1d6-9c04-4826-b329-004c73fb2958\" (UID: \"3173f1d6-9c04-4826-b329-004c73fb2958\") "
Oct 30 21:30:32.780336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.780311    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume" pod "f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" (UID: "f66a8d73-5850-4b6b-88fe-fdbd61b1d9da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:32.780400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.780374    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume" pod "3173f1d6-9c04-4826-b329-004c73fb2958" (UID: "3173f1d6-9c04-4826-b329-004c73fb2958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:32.780758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.780730    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3173f1d6-9c04-4826-b329-004c73fb2958-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:32.780758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.780755    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:32.782096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.782064    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq" pod "f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" (UID: "f66a8d73-5850-4b6b-88fe-fdbd61b1d9da"). InnerVolumeSpecName "kube-api-access-jppbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:32.782230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.782210    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl" pod "3173f1d6-9c04-4826-b329-004c73fb2958" (UID: "3173f1d6-9c04-4826-b329-004c73fb2958"). InnerVolumeSpecName "kube-api-access-wwqtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:32.802545 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:32.802494    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7400-c0203a\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" pod="opendatahub/test-curl-7400-c0203a"
Oct 30 21:30:32.882622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.882528    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppbq\" (UniqueName: \"kubernetes.io/projected/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da-kube-api-access-jppbq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:32.882622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:32.882565    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwqtl\" (UniqueName: \"kubernetes.io/projected/3173f1d6-9c04-4826-b329-004c73fb2958-kube-api-access-wwqtl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:33.230674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.222481    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702a5ec8-413e-4985-8a49-e982c1a8ec8b" path="/var/lib/kubelet/pods/702a5ec8-413e-4985-8a49-e982c1a8ec8b/volumes"
Oct 30 21:30:33.352265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.352208    6695 generic.go:356] "Generic (PLEG): container finished" podID="f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" containerID="4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988" exitCode=0
Oct 30 21:30:33.352668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.352282    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"
Oct 30 21:30:33.352668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.352342    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm" event={"ID":"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da","Type":"ContainerDied","Data":"4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988"}
Oct 30 21:30:33.352668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.352382    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm" event={"ID":"f66a8d73-5850-4b6b-88fe-fdbd61b1d9da","Type":"ContainerDied","Data":"81851cea810baa2ed88c16df462d233ecd42e296dceaa6d4c53a73008b42ffab"}
Oct 30 21:30:33.352668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.352406    6695 scope.go:122] "RemoveContainer" containerID="4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988"
Oct 30 21:30:33.355285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.354867    6695 generic.go:356] "Generic (PLEG): container finished" podID="3173f1d6-9c04-4826-b329-004c73fb2958" containerID="d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065" exitCode=0
Oct 30 21:30:33.355285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.354960    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk" event={"ID":"3173f1d6-9c04-4826-b329-004c73fb2958","Type":"ContainerDied","Data":"d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065"}
Oct 30 21:30:33.355285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.355011    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk" event={"ID":"3173f1d6-9c04-4826-b329-004c73fb2958","Type":"ContainerDied","Data":"9cd6939694014564e24370ae1b4fd1676ea7668ed9cbbd8f786639fbc4497ad0"}
Oct 30 21:30:33.355285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.355056    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"
Oct 30 21:30:33.366432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.366414    6695 scope.go:122] "RemoveContainer" containerID="4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988"
Oct 30 21:30:33.366851 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:33.366813    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988\": container with ID starting with 4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988 not found: ID does not exist" containerID="4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988"
Oct 30 21:30:33.366924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.366852    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988"} err="failed to get container status \"4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988\": rpc error: code = NotFound desc = could not find container \"4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988\": container with ID starting with 4c1ebe28822f5d25d09827635d1ed9cf4d751b2e60b8709739e4be263869e988 not found: ID does not exist"
Oct 30 21:30:33.366924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.366878    6695 scope.go:122] "RemoveContainer" containerID="d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065"
Oct 30 21:30:33.378468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.378427    6695 scope.go:122] "RemoveContainer" containerID="d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065"
Oct 30 21:30:33.378856 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:30:33.378823    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065\": container with ID starting with d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065 not found: ID does not exist" containerID="d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065"
Oct 30 21:30:33.378918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.378865    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065"} err="failed to get container status \"d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065\": rpc error: code = NotFound desc = could not find container \"d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065\": container with ID starting with d657a1586e62d5de81ddd39919e7766d06cd6e99939e0b39106a1d2dbd27f065 not found: ID does not exist"
Oct 30 21:30:33.404392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.404363    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"]
Oct 30 21:30:33.425628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.417409    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c6dab8-6594498f-5ggjm"]
Oct 30 21:30:33.465628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.464024    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"]
Oct 30 21:30:33.469702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.469675    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c6dab8-75bfb75d8c-fxkwk"]
Oct 30 21:30:33.561872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.561847    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:33.592891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.592860    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca\") pod \"be8366fb-2166-4717-9f0b-333d7851b6f7\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") "
Oct 30 21:30:33.593079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.592936    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca\") pod \"be8366fb-2166-4717-9f0b-333d7851b6f7\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") "
Oct 30 21:30:33.593079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.593068    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb\" (UniqueName: \"kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb\") pod \"be8366fb-2166-4717-9f0b-333d7851b6f7\" (UID: \"be8366fb-2166-4717-9f0b-333d7851b6f7\") "
Oct 30 21:30:33.593280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.593259    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca" pod "be8366fb-2166-4717-9f0b-333d7851b6f7" (UID: "be8366fb-2166-4717-9f0b-333d7851b6f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:33.593768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.593742    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:33.595215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.595193    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca" pod "be8366fb-2166-4717-9f0b-333d7851b6f7" (UID: "be8366fb-2166-4717-9f0b-333d7851b6f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:33.595551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.595530    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb" pod "be8366fb-2166-4717-9f0b-333d7851b6f7" (UID: "be8366fb-2166-4717-9f0b-333d7851b6f7"). InnerVolumeSpecName "kube-api-access-w8jtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:33.695359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.695324    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be8366fb-2166-4717-9f0b-333d7851b6f7-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:33.695359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.695355    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jtb\" (UniqueName: \"kubernetes.io/projected/be8366fb-2166-4717-9f0b-333d7851b6f7-kube-api-access-w8jtb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:33.762008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.761970    6695 patch_prober.go:29] interesting pod/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:30:33.762195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:33.762037    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 21:30:34.364203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:34.364171    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"
Oct 30 21:30:34.364676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:34.364180    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j" event={"ID":"be8366fb-2166-4717-9f0b-333d7851b6f7","Type":"ContainerDied","Data":"cca280a6254762247599b268a4538581ea0e8799fb2ff5b57071a1130b26b0b0"}
Oct 30 21:30:34.364676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:34.364309    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca280a6254762247599b268a4538581ea0e8799fb2ff5b57071a1130b26b0b0"
Oct 30 21:30:34.761872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:34.761790    6695 patch_prober.go:29] interesting pod/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:30:34.761872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:34.761844    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 21:30:35.247715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:35.247678    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3173f1d6-9c04-4826-b329-004c73fb2958" path="/var/lib/kubelet/pods/3173f1d6-9c04-4826-b329-004c73fb2958/volumes"
Oct 30 21:30:35.248253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:35.248230    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" path="/var/lib/kubelet/pods/f66a8d73-5850-4b6b-88fe-fdbd61b1d9da/volumes"
Oct 30 21:30:35.762464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:35.762420    6695 patch_prober.go:29] interesting pod/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:30:35.762965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:35.762499    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 21:30:36.766545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:36.766511    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:30:36.766970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:36.766951    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:30:36.767594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:36.767578    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:30:37.030378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:37.030337    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"]
Oct 30 21:30:37.036445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:37.036414    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3a87881-kz4x2"]
Oct 30 21:30:37.246114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:37.246070    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9a333b-d0e9-4727-bb9d-37682cd189ae" path="/var/lib/kubelet/pods/6e9a333b-d0e9-4727-bb9d-37682cd189ae/volumes"
Oct 30 21:30:40.028347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:40.028309    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"]
Oct 30 21:30:40.038723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:40.038692    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f8581f-wcnk6"]
Oct 30 21:30:40.193828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:40.193787    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"]
Oct 30 21:30:41.215208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:41.215166    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6359677-6168-4783-9c33-494489293799" path="/var/lib/kubelet/pods/f6359677-6168-4783-9c33-494489293799/volumes"
Oct 30 21:30:45.194960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.194896    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv" podUID="c15cff4a-47a0-4537-b21b-5c556b18120e" containerName="payload-processing" containerID="cri-o://5f9179c54fb7bbb1bbe5e11667e87f84cf8e95033f45b4fe8025afcdda256574" gracePeriod=25
Oct 30 21:30:45.493591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.493547    6695 generic.go:356] "Generic (PLEG): container finished" podID="c15cff4a-47a0-4537-b21b-5c556b18120e" containerID="5f9179c54fb7bbb1bbe5e11667e87f84cf8e95033f45b4fe8025afcdda256574" exitCode=0
Oct 30 21:30:45.493785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.493707    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv" event={"ID":"c15cff4a-47a0-4537-b21b-5c556b18120e","Type":"ContainerDied","Data":"5f9179c54fb7bbb1bbe5e11667e87f84cf8e95033f45b4fe8025afcdda256574"}
Oct 30 21:30:45.608653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.608623    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:30:45.704656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.704615    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl\" (UniqueName: \"kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl\") pod \"c15cff4a-47a0-4537-b21b-5c556b18120e\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") "
Oct 30 21:30:45.704847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.704695    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume\") pod \"c15cff4a-47a0-4537-b21b-5c556b18120e\" (UID: \"c15cff4a-47a0-4537-b21b-5c556b18120e\") "
Oct 30 21:30:45.705442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.705406    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume" pod "c15cff4a-47a0-4537-b21b-5c556b18120e" (UID: "c15cff4a-47a0-4537-b21b-5c556b18120e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:45.707508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.707466    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl" pod "c15cff4a-47a0-4537-b21b-5c556b18120e" (UID: "c15cff4a-47a0-4537-b21b-5c556b18120e"). InnerVolumeSpecName "kube-api-access-8zqrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:45.806777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.806742    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqrl\" (UniqueName: \"kubernetes.io/projected/c15cff4a-47a0-4537-b21b-5c556b18120e-kube-api-access-8zqrl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:45.806777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:45.806772    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15cff4a-47a0-4537-b21b-5c556b18120e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:30:46.505490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:46.505454    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"
Oct 30 21:30:46.506016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:46.505488    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv" event={"ID":"c15cff4a-47a0-4537-b21b-5c556b18120e","Type":"ContainerDied","Data":"ed1fcd90e04d921489264117da277e719789a700d213ff23bf7b51b1e60fc66a"}
Oct 30 21:30:46.506016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:46.505568    6695 scope.go:122] "RemoveContainer" containerID="5f9179c54fb7bbb1bbe5e11667e87f84cf8e95033f45b4fe8025afcdda256574"
Oct 30 21:30:46.561198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:46.561160    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"]
Oct 30 21:30:46.569537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:46.569505    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ab5774-565d7fb9f-gvppv"]
Oct 30 21:30:47.217036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:47.217002    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15cff4a-47a0-4537-b21b-5c556b18120e" path="/var/lib/kubelet/pods/c15cff4a-47a0-4537-b21b-5c556b18120e/volumes"
Oct 30 21:30:49.343280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:49.343242    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"]
Oct 30 21:30:49.347295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:49.347269    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f2f4ebe-s4h4g"]
Oct 30 21:30:51.263149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:51.263092    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" path="/var/lib/kubelet/pods/526e7c21-fdfd-43b2-aef5-ec7d1c1b73df/volumes"
Oct 30 21:30:57.027096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:57.027058    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"]
Oct 30 21:30:57.037676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:57.037645    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c30c14b3-rm7mf"]
Oct 30 21:30:57.215620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:30:57.215575    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bebea0c-a750-4a7e-87b6-0bab7f90ecb1" path="/var/lib/kubelet/pods/4bebea0c-a750-4a7e-87b6-0bab7f90ecb1/volumes"
Oct 30 21:31:01.030836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:01.030798    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"]
Oct 30 21:31:01.041522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:01.041480    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"]
Oct 30 21:31:01.045270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:01.045242    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6d62e33-bjd5n"]
Oct 30 21:31:01.048864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:01.048839    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce1c589-cxxvr"]
Oct 30 21:31:01.078130 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:31:01.078089    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:31:01.219471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:01.219435    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eff7c38-5dc0-4f75-b722-6f36be69687c" path="/var/lib/kubelet/pods/1eff7c38-5dc0-4f75-b722-6f36be69687c/volumes"
Oct 30 21:31:01.220376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:01.220349    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8781ebe8-db2b-4577-b22c-6a16447a6586" path="/var/lib/kubelet/pods/8781ebe8-db2b-4577-b22c-6a16447a6586/volumes"
Oct 30 21:31:02.139754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:02.139710    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"]
Oct 30 21:31:02.151549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:02.151499    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c6dab8-phzbx"]
Oct 30 21:31:02.708658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:02.708622    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"]
Oct 30 21:31:02.719841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:02.719801    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab5774-g5n6j"]
Oct 30 21:31:03.214706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:03.214676    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8366fb-2166-4717-9f0b-333d7851b6f7" path="/var/lib/kubelet/pods/be8366fb-2166-4717-9f0b-333d7851b6f7/volumes"
Oct 30 21:31:03.215191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:03.215177    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d8bdde-4f48-4479-ade4-961b6be7001a" path="/var/lib/kubelet/pods/c2d8bdde-4f48-4479-ade4-961b6be7001a/volumes"
Oct 30 21:31:05.033405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:05.033366    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"]
Oct 30 21:31:05.044967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:05.044938    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a33decec-rlwwc"]
Oct 30 21:31:05.214930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:05.214896    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1562005-296e-4cb8-8870-7e7dec936f6a" path="/var/lib/kubelet/pods/c1562005-296e-4cb8-8870-7e7dec936f6a/volumes"
Oct 30 21:31:07.643725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.643685    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"]
Oct 30 21:31:07.644837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.644804    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" containerName="payload-processing"
Oct 30 21:31:07.645096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645070    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" containerName="payload-processing"
Oct 30 21:31:07.645214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645121    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c15cff4a-47a0-4537-b21b-5c556b18120e" containerName="payload-processing"
Oct 30 21:31:07.645214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645129    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15cff4a-47a0-4537-b21b-5c556b18120e" containerName="payload-processing"
Oct 30 21:31:07.645214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645144    6695 cpu_manager.go:414] "containerMap: removing container" podUID="be8366fb-2166-4717-9f0b-333d7851b6f7" containerName="revoke-keys"
Oct 30 21:31:07.645214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645149    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8366fb-2166-4717-9f0b-333d7851b6f7" containerName="revoke-keys"
Oct 30 21:31:07.645214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645183    6695 cpu_manager.go:414] "containerMap: removing container" podUID="526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" containerName="revoke-keys"
Oct 30 21:31:07.645214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645191    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" containerName="revoke-keys"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645226    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3173f1d6-9c04-4826-b329-004c73fb2958" containerName="payload-pre-processing"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645233    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3173f1d6-9c04-4826-b329-004c73fb2958" containerName="payload-pre-processing"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645406    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3173f1d6-9c04-4826-b329-004c73fb2958" containerName="payload-pre-processing"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645423    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66a8d73-5850-4b6b-88fe-fdbd61b1d9da" containerName="payload-processing"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645434    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15cff4a-47a0-4537-b21b-5c556b18120e" containerName="payload-processing"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645443    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8366fb-2166-4717-9f0b-333d7851b6f7" containerName="revoke-keys"
Oct 30 21:31:07.645475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.645455    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="526e7c21-fdfd-43b2-aef5-ec7d1c1b73df" containerName="revoke-keys"
Oct 30 21:31:07.649150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.649127    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.663887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.663863    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-229795-openshift-default-dockercfg-bf2pn\""
Oct 30 21:31:07.681635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.681581    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"]
Oct 30 21:31:07.747363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.747320    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.747679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.747657    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.747868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.747849    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.747956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.747894    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.747956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.747939    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.748067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.748001    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.748117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.748100    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.748168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.748140    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.748225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.748192    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sjk\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.850952    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851016    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851058    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851115    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851205    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851258    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851325    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sjk\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851484    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.851531    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.852179    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.855009    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.855677    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.856636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.856220    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.857469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.856909    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.873625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.871389    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.873625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.871711    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.894702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.894587    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sjk\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.900684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.899079    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token\") pod \"e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") " pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:07.975694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:07.974360    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:08.473825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.473798    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"]
Oct 30 21:31:08.476914 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:31:08.476886    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83a9b10_e409_4d52_b108_41fe7289ea5f.slice/crio-7025f94bf2d4e4bf6881d7a26c2badeb7b57284a7935910bbde558fca39e7531 WatchSource:0}: Error finding container 7025f94bf2d4e4bf6881d7a26c2badeb7b57284a7935910bbde558fca39e7531: Status 404 returned error can't find the container with id 7025f94bf2d4e4bf6881d7a26c2badeb7b57284a7935910bbde558fca39e7531
Oct 30 21:31:08.479535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.479498    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:08.479665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.479569    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:08.479665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.479598    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:08.735556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.735454    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" event={"ID":"a83a9b10-e409-4d52-b108-41fe7289ea5f","Type":"ContainerStarted","Data":"63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"}
Oct 30 21:31:08.735556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.735520    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" event={"ID":"a83a9b10-e409-4d52-b108-41fe7289ea5f","Type":"ContainerStarted","Data":"7025f94bf2d4e4bf6881d7a26c2badeb7b57284a7935910bbde558fca39e7531"}
Oct 30 21:31:08.777645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.777575    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podStartSLOduration=1.777560538 podStartE2EDuration="1.777560538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:08.776050731 +0000 UTC m=+1251.698509029" watchObservedRunningTime="2026-10-30 21:31:08.777560538 +0000 UTC m=+1251.700018837"
Oct 30 21:31:08.987676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:08.975706    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:09.050629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:09.049658    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"]
Oct 30 21:31:09.061614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:09.053910    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43248925-xrz9p"]
Oct 30 21:31:09.246015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:09.243920    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688b26a3-59ab-412f-ab65-ed2db67d7083" path="/var/lib/kubelet/pods/688b26a3-59ab-412f-ab65-ed2db67d7083/volumes"
Oct 30 21:31:09.993629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:09.993190    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:10.756682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:10.756651    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:10.757666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:10.757644    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:10.861881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:10.861852    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"]
Oct 30 21:31:10.862074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:10.862050    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy" containerID="cri-o://a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042" gracePeriod=30
Oct 30 21:31:11.042337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.042300    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"]
Oct 30 21:31:11.045987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.045957    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b7de24-7mn7x"]
Oct 30 21:31:11.096803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.096762    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"]
Oct 30 21:31:11.101717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.101687    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.121564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.121070    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c027ae-openshift-default-dockercfg-s6hb4\""
Oct 30 21:31:11.130527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.124208    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"]
Oct 30 21:31:11.220070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.219990    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc94cba4-80b7-4a90-84f5-c9be9494b031" path="/var/lib/kubelet/pods/dc94cba4-80b7-4a90-84f5-c9be9494b031/volumes"
Oct 30 21:31:11.302785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.298711    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.319298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.319264    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.320169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.320145    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.320639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.320537    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.321266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.321246    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.326066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.326044    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.342362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.342329    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.343025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.342914    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.343210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.343190    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tszdg\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.445827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.445787    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.446102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.446076    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszdg\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.446424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.446403    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.446726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.446705    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.446930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.446902    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.447133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.447117    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.447259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.447244    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.447441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.447428    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.447574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.447562    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.453397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.453372    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.453960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.453926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.458300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.455903    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.458300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.456742    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.459368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.459266    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.473114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.472083    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.479632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.478263    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.481680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.481571    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszdg\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.490074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.489998    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy\") pod \"e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") " pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.721728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.721596    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:11.812359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.811703    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"]
Oct 30 21:31:11.817546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.817514    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.835700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.833831    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:11.842726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.839765    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:11.842726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.840768    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"]
Oct 30 21:31:11.866016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.865909    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j9t7\" (UniqueName: \"kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.866388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.866356    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.866631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.866556    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.920708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.920673    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:31:11.920708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.920673    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:11.969491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.969454    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j9t7\" (UniqueName: \"kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.969693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.969513    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.969693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.969558    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.970117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.970089    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.972047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.971987    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:11.989368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:11.989340    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j9t7\" (UniqueName: \"kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7\") pod \"maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:12.064323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.064085    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"]
Oct 30 21:31:12.100920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.100263    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:12.100920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.100362    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:12.100920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.100405    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:12.156637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.155955    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:12.488036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.487948    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"]
Oct 30 21:31:12.496169 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:31:12.496001    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c91fda_37dc_4fe9_8756_154804c39cc5.slice/crio-e2d55dd7b2382468b59c6956198ab3042a8bfd7630757e914b7dc2b2dbfc3968 WatchSource:0}: Error finding container e2d55dd7b2382468b59c6956198ab3042a8bfd7630757e914b7dc2b2dbfc3968: Status 404 returned error can't find the container with id e2d55dd7b2382468b59c6956198ab3042a8bfd7630757e914b7dc2b2dbfc3968
Oct 30 21:31:12.776673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.776634    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" event={"ID":"8002939f-fa35-4c34-bbfd-da44de646313","Type":"ContainerStarted","Data":"4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"}
Oct 30 21:31:12.776887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.776701    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" event={"ID":"8002939f-fa35-4c34-bbfd-da44de646313","Type":"ContainerStarted","Data":"343b9c5a707363c299b457207b215d24cfaf18aea44075d8e9d8e70501df34cd"}
Oct 30 21:31:12.778300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.778273    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g" event={"ID":"e9c91fda-37dc-4fe9-8756-154804c39cc5","Type":"ContainerStarted","Data":"e2d55dd7b2382468b59c6956198ab3042a8bfd7630757e914b7dc2b2dbfc3968"}
Oct 30 21:31:12.820984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:12.820915    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podStartSLOduration=1.8208952919999999 podStartE2EDuration="1.820895292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:12.817399994 +0000 UTC m=+1255.739858297" watchObservedRunningTime="2026-10-30 21:31:12.820895292 +0000 UTC m=+1255.743353591"
Oct 30 21:31:13.350159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.350125    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"]
Oct 30 21:31:13.354517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.354491    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.368630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.368588    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-229795\""
Oct 30 21:31:13.371720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.371683    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-229795-dockercfg-cx74b\""
Oct 30 21:31:13.372550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.372523    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"]
Oct 30 21:31:13.647432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.647333    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.647626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.647459    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2s7\" (UniqueName: \"kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7\") pod \"payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.734504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.734307    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:13.782331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.782300    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:13.794540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.794508    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g" event={"ID":"e9c91fda-37dc-4fe9-8756-154804c39cc5","Type":"ContainerStarted","Data":"982712d3616417f522f98dd3fdfee019487341d9ce09616e969127768eb363cd"}
Oct 30 21:31:13.795243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.795220    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:13.796122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.796101    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:13.870993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.870958    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.871172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.871024    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2s7\" (UniqueName: \"kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7\") pod \"payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.871856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.871828    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.889971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.889941    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2s7\" (UniqueName: \"kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7\") pod \"payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:13.969225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:13.969133    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:14.356705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.356672    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"]
Oct 30 21:31:14.360895 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:31:14.360855    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aef0c41_292d_4096_9fea_2765bc2c3983.slice/crio-9bcaddeec8db8bc0f63662aba01d3cb90605f0cfd8eab39c63a4965dcbf0a421 WatchSource:0}: Error finding container 9bcaddeec8db8bc0f63662aba01d3cb90605f0cfd8eab39c63a4965dcbf0a421: Status 404 returned error can't find the container with id 9bcaddeec8db8bc0f63662aba01d3cb90605f0cfd8eab39c63a4965dcbf0a421
Oct 30 21:31:14.804806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.804773    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6" event={"ID":"1aef0c41-292d-4096-9fea-2765bc2c3983","Type":"ContainerStarted","Data":"be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66"}
Oct 30 21:31:14.805021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.804831    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6" event={"ID":"1aef0c41-292d-4096-9fea-2765bc2c3983","Type":"ContainerStarted","Data":"9bcaddeec8db8bc0f63662aba01d3cb90605f0cfd8eab39c63a4965dcbf0a421"}
Oct 30 21:31:14.805021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.804863    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:14.806487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.806461    6695 generic.go:356] "Generic (PLEG): container finished" podID="e9c91fda-37dc-4fe9-8756-154804c39cc5" containerID="982712d3616417f522f98dd3fdfee019487341d9ce09616e969127768eb363cd" exitCode=0
Oct 30 21:31:14.806617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.806528    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g" event={"ID":"e9c91fda-37dc-4fe9-8756-154804c39cc5","Type":"ContainerDied","Data":"982712d3616417f522f98dd3fdfee019487341d9ce09616e969127768eb363cd"}
Oct 30 21:31:14.849312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:14.849247    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6" podStartSLOduration=1.849226815 podStartE2EDuration="1.849226815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:14.843898349 +0000 UTC m=+1257.766356648" watchObservedRunningTime="2026-10-30 21:31:14.849226815 +0000 UTC m=+1257.771685115"
Oct 30 21:31:15.047630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.042948    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:15.064660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.064512    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"]
Oct 30 21:31:15.068554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.068518    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b4ea00-kxnsc"]
Oct 30 21:31:15.205872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.205833    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca\") pod \"e9c91fda-37dc-4fe9-8756-154804c39cc5\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") "
Oct 30 21:31:15.206048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.205997    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca\") pod \"e9c91fda-37dc-4fe9-8756-154804c39cc5\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") "
Oct 30 21:31:15.206128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.206112    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7\" (UniqueName: \"kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7\") pod \"e9c91fda-37dc-4fe9-8756-154804c39cc5\" (UID: \"e9c91fda-37dc-4fe9-8756-154804c39cc5\") "
Oct 30 21:31:15.206193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.206151    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca" pod "e9c91fda-37dc-4fe9-8756-154804c39cc5" (UID: "e9c91fda-37dc-4fe9-8756-154804c39cc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:15.206823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.206796    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.208315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.208281    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca" pod "e9c91fda-37dc-4fe9-8756-154804c39cc5" (UID: "e9c91fda-37dc-4fe9-8756-154804c39cc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:15.208578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.208556    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7" pod "e9c91fda-37dc-4fe9-8756-154804c39cc5" (UID: "e9c91fda-37dc-4fe9-8756-154804c39cc5"). InnerVolumeSpecName "kube-api-access-2j9t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.217070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.217044    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a98c7fc3-b122-4bff-bb9d-1c8471f0223e" path="/var/lib/kubelet/pods/a98c7fc3-b122-4bff-bb9d-1c8471f0223e/volumes"
Oct 30 21:31:15.307947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.307916    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c91fda-37dc-4fe9-8756-154804c39cc5-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.307947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.307944    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j9t7\" (UniqueName: \"kubernetes.io/projected/e9c91fda-37dc-4fe9-8756-154804c39cc5-kube-api-access-2j9t7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.817208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.817175    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"
Oct 30 21:31:15.817711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.817205    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g" event={"ID":"e9c91fda-37dc-4fe9-8756-154804c39cc5","Type":"ContainerDied","Data":"e2d55dd7b2382468b59c6956198ab3042a8bfd7630757e914b7dc2b2dbfc3968"}
Oct 30 21:31:15.817711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:15.817274    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2d55dd7b2382468b59c6956198ab3042a8bfd7630757e914b7dc2b2dbfc3968"
Oct 30 21:31:16.153681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.153653    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:31:16.342852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.342813    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.342883    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.342994    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.343033    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.343080    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.343115    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.343195    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.343382    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.343905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.343453    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9\") pod \"f3f502ae-377f-48ef-b68f-21d116b0d87c\" (UID: \"f3f502ae-377f-48ef-b68f-21d116b0d87c\") "
Oct 30 21:31:16.344778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.344752    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:16.344983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.344964    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:16.345587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.345564    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:16.346128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.346101    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:16.346230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.346112    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:16.346230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.346183    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:16.347778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.347752    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:16.347778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.347760    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "kube-api-access-xrgr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:16.347778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.347776    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token" pod "f3f502ae-377f-48ef-b68f-21d116b0d87c" (UID: "f3f502ae-377f-48ef-b68f-21d116b0d87c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445435    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445470    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445479    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f502ae-377f-48ef-b68f-21d116b0d87c-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445488    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445497    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445505    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445515    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrgr9\" (UniqueName: \"kubernetes.io/projected/f3f502ae-377f-48ef-b68f-21d116b0d87c-kube-api-access-xrgr9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445524    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f502ae-377f-48ef-b68f-21d116b0d87c-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.445527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.445532    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f502ae-377f-48ef-b68f-21d116b0d87c-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.827502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.827463    6695 generic.go:356] "Generic (PLEG): container finished" podID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerID="a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042" exitCode=0
Oct 30 21:31:16.827999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.827522    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"
Oct 30 21:31:16.827999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.827539    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" event={"ID":"f3f502ae-377f-48ef-b68f-21d116b0d87c","Type":"ContainerDied","Data":"a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042"}
Oct 30 21:31:16.827999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.827637    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv" event={"ID":"f3f502ae-377f-48ef-b68f-21d116b0d87c","Type":"ContainerDied","Data":"3876a0d82e0e3a09cadaeda6e38e9e5c14a619c978d2b6344d43e69b0bdff152"}
Oct 30 21:31:16.827999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.827664    6695 scope.go:122] "RemoveContainer" containerID="a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042"
Oct 30 21:31:16.845888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.845862    6695 scope.go:122] "RemoveContainer" containerID="586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"
Oct 30 21:31:16.856879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.856854    6695 scope.go:122] "RemoveContainer" containerID="a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042"
Oct 30 21:31:16.857200 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:31:16.857178    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042\": container with ID starting with a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042 not found: ID does not exist" containerID="a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042"
Oct 30 21:31:16.857277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.857210    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042"} err="failed to get container status \"a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042\": rpc error: code = NotFound desc = could not find container \"a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042\": container with ID starting with a2d8923649d68c5c2edbf4b9419b49fb2e35efd7bddc4af9a16e2920c0e42042 not found: ID does not exist"
Oct 30 21:31:16.857277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.857229    6695 scope.go:122] "RemoveContainer" containerID="586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"
Oct 30 21:31:16.857502 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:31:16.857481    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6\": container with ID starting with 586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6 not found: ID does not exist" containerID="586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"
Oct 30 21:31:16.857556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.857508    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6"} err="failed to get container status \"586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6\": rpc error: code = NotFound desc = could not find container \"586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6\": container with ID starting with 586492a2cfd0afb6a1f47850c46d509c488a62fd71c03f8246e12a10afcc78e6 not found: ID does not exist"
Oct 30 21:31:16.903428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.903391    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"]
Oct 30 21:31:16.927586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:16.927547    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ab5774-openshift-default-5b7dc45fbf-hbbgv"]
Oct 30 21:31:17.215496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:17.215421    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" path="/var/lib/kubelet/pods/f3f502ae-377f-48ef-b68f-21d116b0d87c/volumes"
Oct 30 21:31:18.132838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:18.132791    6695 scope.go:122] "RemoveContainer" containerID="ea5f94b14d3d4ee3b9771180bdee312bd6d0f790158d1b5392b94eab7e50327e"
Oct 30 21:31:18.143497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:18.143473    6695 scope.go:122] "RemoveContainer" containerID="b1e4899ef26d2942e06831b28534bb9616486a06852835c2d9bbd7445c31494a"
Oct 30 21:31:21.029454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:21.029418    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"]
Oct 30 21:31:21.040403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:21.040368    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"]
Oct 30 21:31:21.044488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:21.044460    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c638681d-cbq6k"]
Oct 30 21:31:21.047515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:21.047493    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30e39f1-xg8wf"]
Oct 30 21:31:21.214656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:21.214621    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4b0601-8bf6-41d8-a382-a70121071475" path="/var/lib/kubelet/pods/4f4b0601-8bf6-41d8-a382-a70121071475/volumes"
Oct 30 21:31:21.215102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:21.215088    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad1d93c-7486-4e99-8a28-108dbf4b659d" path="/var/lib/kubelet/pods/dad1d93c-7486-4e99-8a28-108dbf4b659d/volumes"
Oct 30 21:31:25.820682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:25.820594    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:26.087121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:26.085865    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"]
Oct 30 21:31:26.091698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:26.091669    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"]
Oct 30 21:31:26.097439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:26.097404    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9de0cb0f-r5hpt"]
Oct 30 21:31:26.100207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:26.100181    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dcaa342-pmdd5"]
Oct 30 21:31:27.032179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:27.032145    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"]
Oct 30 21:31:27.043129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:27.043099    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31b8df8-vf7w5"]
Oct 30 21:31:27.214840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:27.214797    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b758a0aa-f78e-490b-8069-71758a4ffdd0" path="/var/lib/kubelet/pods/b758a0aa-f78e-490b-8069-71758a4ffdd0/volumes"
Oct 30 21:31:27.215395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:27.215374    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4f7d23-1179-459c-a97c-5ae1420f0b85" path="/var/lib/kubelet/pods/bd4f7d23-1179-459c-a97c-5ae1420f0b85/volumes"
Oct 30 21:31:27.215899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:27.215879    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3fab7bd-49b9-41e5-ba47-d406e826074e" path="/var/lib/kubelet/pods/c3fab7bd-49b9-41e5-ba47-d406e826074e/volumes"
Oct 30 21:31:31.074641 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:31:31.074544    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:31:32.031320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:32.031280    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"]
Oct 30 21:31:32.042121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:32.042087    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925dad17-zmsgz"]
Oct 30 21:31:33.214638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:33.214589    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5b6f9b-3973-4d62-b9c0-1506672b7927" path="/var/lib/kubelet/pods/ad5b6f9b-3973-4d62-b9c0-1506672b7927/volumes"
Oct 30 21:31:41.710065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:41.710022    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"]
Oct 30 21:31:42.314598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:42.314556    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"]
Oct 30 21:31:42.394628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:42.390171    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe10e0c0-9bqrd"]
Oct 30 21:31:43.219726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:43.219688    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4813512f-df13-40c1-aa89-c1f9d07840d7" path="/var/lib/kubelet/pods/4813512f-df13-40c1-aa89-c1f9d07840d7/volumes"
Oct 30 21:31:44.065658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:44.060658    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"]
Oct 30 21:31:44.069215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:44.069165    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55abbb3e-9t5qr"]
Oct 30 21:31:45.249987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:45.249950    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6fdd92-d7a6-4b6e-85c5-04a94152335b" path="/var/lib/kubelet/pods/4d6fdd92-d7a6-4b6e-85c5-04a94152335b/volumes"
Oct 30 21:31:46.714721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:46.714672    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6" podUID="1aef0c41-292d-4096-9fea-2765bc2c3983" containerName="payload-pre-processing" containerID="cri-o://be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66" gracePeriod=25
Oct 30 21:31:47.265474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.265343    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:47.287947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.287913    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume\") pod \"1aef0c41-292d-4096-9fea-2765bc2c3983\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") "
Oct 30 21:31:47.288178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.288165    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7\" (UniqueName: \"kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7\") pod \"1aef0c41-292d-4096-9fea-2765bc2c3983\" (UID: \"1aef0c41-292d-4096-9fea-2765bc2c3983\") "
Oct 30 21:31:47.288403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.288374    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume" pod "1aef0c41-292d-4096-9fea-2765bc2c3983" (UID: "1aef0c41-292d-4096-9fea-2765bc2c3983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:47.288781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.288761    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aef0c41-292d-4096-9fea-2765bc2c3983-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:47.290324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.290294    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7" pod "1aef0c41-292d-4096-9fea-2765bc2c3983" (UID: "1aef0c41-292d-4096-9fea-2765bc2c3983"). InnerVolumeSpecName "kube-api-access-vh2s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:47.469400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.469192    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2s7\" (UniqueName: \"kubernetes.io/projected/1aef0c41-292d-4096-9fea-2765bc2c3983-kube-api-access-vh2s7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:31:47.552057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.551984    6695 generic.go:356] "Generic (PLEG): container finished" podID="1aef0c41-292d-4096-9fea-2765bc2c3983" containerID="be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66" exitCode=0
Oct 30 21:31:47.552842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.552809    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"
Oct 30 21:31:47.568736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.568681    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6" event={"ID":"1aef0c41-292d-4096-9fea-2765bc2c3983","Type":"ContainerDied","Data":"be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66"}
Oct 30 21:31:47.571202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.571154    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6" event={"ID":"1aef0c41-292d-4096-9fea-2765bc2c3983","Type":"ContainerDied","Data":"9bcaddeec8db8bc0f63662aba01d3cb90605f0cfd8eab39c63a4965dcbf0a421"}
Oct 30 21:31:47.571456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.571439    6695 scope.go:122] "RemoveContainer" containerID="be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66"
Oct 30 21:31:47.714998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.714975    6695 scope.go:122] "RemoveContainer" containerID="be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66"
Oct 30 21:31:47.715422 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:31:47.715394    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66\": container with ID starting with be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66 not found: ID does not exist" containerID="be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66"
Oct 30 21:31:47.715492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.715434    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66"} err="failed to get container status \"be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66\": rpc error: code = NotFound desc = could not find container \"be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66\": container with ID starting with be2e6e3bbfbd3f43fa998145a2df2be7eefda9e3084033ca7c8c598639d76d66 not found: ID does not exist"
Oct 30 21:31:47.737435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.737402    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"]
Oct 30 21:31:47.748478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:47.748440    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-6dfb59b7f-z5nf6"]
Oct 30 21:31:49.216203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:49.216159    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aef0c41-292d-4096-9fea-2765bc2c3983" path="/var/lib/kubelet/pods/1aef0c41-292d-4096-9fea-2765bc2c3983/volumes"
Oct 30 21:31:53.969130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:53.969095    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"]
Oct 30 21:31:53.977621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:53.977576    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-342fc9-xhz4g"]
Oct 30 21:31:54.060629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.053048    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"]
Oct 30 21:31:54.070046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.070000    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8498fa7-vnwnz"]
Oct 30 21:31:54.327462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.327371    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"]
Oct 30 21:31:54.328434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328406    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e9c91fda-37dc-4fe9-8756-154804c39cc5" containerName="revoke-keys"
Oct 30 21:31:54.328434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328433    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c91fda-37dc-4fe9-8756-154804c39cc5" containerName="revoke-keys"
Oct 30 21:31:54.328682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328470    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy"
Oct 30 21:31:54.328682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328478    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy"
Oct 30 21:31:54.328682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328495    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy"
Oct 30 21:31:54.328682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328503    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy"
Oct 30 21:31:54.328682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328573    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1aef0c41-292d-4096-9fea-2765bc2c3983" containerName="payload-pre-processing"
Oct 30 21:31:54.328682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328582    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aef0c41-292d-4096-9fea-2765bc2c3983" containerName="payload-pre-processing"
Oct 30 21:31:54.329024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328747    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aef0c41-292d-4096-9fea-2765bc2c3983" containerName="payload-pre-processing"
Oct 30 21:31:54.329024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328765    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c91fda-37dc-4fe9-8756-154804c39cc5" containerName="revoke-keys"
Oct 30 21:31:54.329024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328779    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy"
Oct 30 21:31:54.329024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.328790    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f502ae-377f-48ef-b68f-21d116b0d87c" containerName="istio-proxy"
Oct 30 21:31:54.383345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.383247    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"]
Oct 30 21:31:54.383345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.383344    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.399824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.399794    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3afc5e-openshift-default-dockercfg-rdxf8\""
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.499772    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.499833    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.499887    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.499950    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgtw\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.499994    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.548393    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.548575    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.548760    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.629063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.548912    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653198    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653410    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653492    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653858    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653904    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.653955    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.654022    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgtw\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.654063    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.661120    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.663496    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.663912    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.664105    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.664249    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.677217    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.709962    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.720051    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgtw\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.728189    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo\") pod \"e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") " pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:54.788501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:54.754474    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:55.153197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.153169    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"]
Oct 30 21:31:55.155718 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:31:55.155683    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6259c498_a1b5_4aba_8d29_1fe6a590d24f.slice/crio-1af5f1b281aab579508991334f03b39498646c53dd40f2ae2a486817f2fc3b3e WatchSource:0}: Error finding container 1af5f1b281aab579508991334f03b39498646c53dd40f2ae2a486817f2fc3b3e: Status 404 returned error can't find the container with id 1af5f1b281aab579508991334f03b39498646c53dd40f2ae2a486817f2fc3b3e
Oct 30 21:31:55.158236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.158203    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.158321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.158276    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.158321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.158303    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.216592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.216566    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2ac0a9-4f99-42c4-b546-185fcacf1084" path="/var/lib/kubelet/pods/1b2ac0a9-4f99-42c4-b546-185fcacf1084/volumes"
Oct 30 21:31:55.217224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.217201    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c91fda-37dc-4fe9-8756-154804c39cc5" path="/var/lib/kubelet/pods/e9c91fda-37dc-4fe9-8756-154804c39cc5/volumes"
Oct 30 21:31:55.757991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.757953    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:31:55.758141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.758005    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:31:55.782716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.782678    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" event={"ID":"6259c498-a1b5-4aba-8d29-1fe6a590d24f","Type":"ContainerStarted","Data":"3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"}
Oct 30 21:31:55.782908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.782738    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" event={"ID":"6259c498-a1b5-4aba-8d29-1fe6a590d24f","Type":"ContainerStarted","Data":"1af5f1b281aab579508991334f03b39498646c53dd40f2ae2a486817f2fc3b3e"}
Oct 30 21:31:55.784467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.784443    6695 generic.go:356] "Generic (PLEG): container finished" podID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerID="63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b" exitCode=0
Oct 30 21:31:55.784582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.784515    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" event={"ID":"a83a9b10-e409-4d52-b108-41fe7289ea5f","Type":"ContainerDied","Data":"63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"}
Oct 30 21:31:55.785001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.784981    6695 scope.go:122] "RemoveContainer" containerID="63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"
Oct 30 21:31:55.786422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.786400    6695 generic.go:356] "Generic (PLEG): container finished" podID="8002939f-fa35-4c34-bbfd-da44de646313" containerID="4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d" exitCode=0
Oct 30 21:31:55.786536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.786432    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" event={"ID":"8002939f-fa35-4c34-bbfd-da44de646313","Type":"ContainerDied","Data":"4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"}
Oct 30 21:31:55.786653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.786594    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.786727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.786711    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.786786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.786769    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.786931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.786909    6695 scope.go:122] "RemoveContainer" containerID="4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"
Oct 30 21:31:55.788243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.788216    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.788349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.788294    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.788349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.788329    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:55.830978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:55.830932    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podStartSLOduration=1.8309183770000002 podStartE2EDuration="1.830918377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:55.829246151 +0000 UTC m=+1298.751704448" watchObservedRunningTime="2026-10-30 21:31:55.830918377 +0000 UTC m=+1298.753376674"
Oct 30 21:31:56.756853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.756798    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:56.765433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.765402    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:56.796807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.796771    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" event={"ID":"a83a9b10-e409-4d52-b108-41fe7289ea5f","Type":"ContainerStarted","Data":"e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba"}
Oct 30 21:31:56.799301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.799257    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" event={"ID":"8002939f-fa35-4c34-bbfd-da44de646313","Type":"ContainerStarted","Data":"018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e"}
Oct 30 21:31:56.799461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.799348    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:56.800881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.800854    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:31:56.975045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.975010    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:31:56.975282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.975259    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:31:56.975334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:56.975306    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:31:57.733090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:57.733061    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:57.733579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:57.733552    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:31:57.733818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:57.733789    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:31:58.007732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.001447    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:31:58.007732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.001506    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:31:58.728629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.726216    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:31:58.728629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.726286    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:31:58.794997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.794958    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:31:58.976684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.976651    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:31:58.976946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:58.976925    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:31:59.731114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:59.731072    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:31:59.731761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:59.731725    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:31:59.975336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:59.975301    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:31:59.975486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:31:59.975350    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:32:00.370148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.370102    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"]
Oct 30 21:32:00.373852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.373825    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.404292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.404261    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"]
Oct 30 21:32:00.404479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.404391    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3afc5e-dockercfg-tb8bs\""
Oct 30 21:32:00.404479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.404444    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3afc5e\""
Oct 30 21:32:00.405696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.405668    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.405856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.405837    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqhx\" (UniqueName: \"kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.477978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.477924    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"]
Oct 30 21:32:00.481855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.481829    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"]
Oct 30 21:32:00.482062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.481924    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.514648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.511328    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqhx\" (UniqueName: \"kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.514648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.511395    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.514648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.511620    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crwj\" (UniqueName: \"kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj\") pod \"payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.514648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.511668    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.566766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.553287    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.613150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.613107    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crwj\" (UniqueName: \"kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj\") pod \"payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.623006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.622916    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.673846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.673337    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqhx\" (UniqueName: \"kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.696628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.682160    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.696940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.694863    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:00.760232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.754806    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crwj\" (UniqueName: \"kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj\") pod \"payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.794634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.781446    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:32:00.794634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.781514    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:32:00.806636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.803574    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:00.982741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.979014    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:32:00.982741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:00.979074    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:32:01.222059 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:01.221956    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:32:01.589667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:01.580129    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"]
Oct 30 21:32:01.589667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:01.580379    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" containerID="cri-o://f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489" gracePeriod=30
Oct 30 21:32:01.731130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:01.731042    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:32:01.731392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:01.731363    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:32:02.011693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:01.992271    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"]
Oct 30 21:32:02.011693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.005123    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:32:02.011693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.005178    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:32:02.078732 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:02.070768    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3c3028_b40a_4e77_8826_59b7efe0f6e4.slice/crio-8daf1f8ef19fcf0ff82cc9cbe49fdf7752abf6fab67d935ddf3cc589a670d88f WatchSource:0}: Error finding container 8daf1f8ef19fcf0ff82cc9cbe49fdf7752abf6fab67d935ddf3cc589a670d88f: Status 404 returned error can't find the container with id 8daf1f8ef19fcf0ff82cc9cbe49fdf7752abf6fab67d935ddf3cc589a670d88f
Oct 30 21:32:02.134627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.130353    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"]
Oct 30 21:32:02.723154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.723114    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:32:02.723441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.723397    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:32:02.983689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.977050    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:32:02.983689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:02.977119    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:32:03.046629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.040793    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"]
Oct 30 21:32:03.074714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.074677    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"]
Oct 30 21:32:03.074931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.074883    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.091359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.091258    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0a1ee3-openshift-default-dockercfg-f69wk\""
Oct 30 21:32:03.197015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.196948    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.197382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.197355    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.197647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.197627    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5sl\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.197811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.197794    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.197920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.197903    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.198033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.198019    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.198145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.198132    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.200400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.198359    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.200721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.200697    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.359409    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383022    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383100    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383228    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383283    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5sl\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383338    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383367    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383408    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.383451    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.360173    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.385162    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.385433    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.386012    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.388393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.387168    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.425637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.415825    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm" event={"ID":"ff3c3028-b40a-4e77-8826-59b7efe0f6e4","Type":"ContainerStarted","Data":"820d4982ed2dd68dd253183beaba3187637b7f349e771d6ecd3cdd732615173e"}
Oct 30 21:32:03.425637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.415916    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm" event={"ID":"ff3c3028-b40a-4e77-8826-59b7efe0f6e4","Type":"ContainerStarted","Data":"8daf1f8ef19fcf0ff82cc9cbe49fdf7752abf6fab67d935ddf3cc589a670d88f"}
Oct 30 21:32:03.425637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.417151    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:03.449627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.439161    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.449627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.440922    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx" event={"ID":"3f853e10-748b-4fca-acab-bd324a88b1f1","Type":"ContainerStarted","Data":"10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc"}
Oct 30 21:32:03.449627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.440987    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx" event={"ID":"3f853e10-748b-4fca-acab-bd324a88b1f1","Type":"ContainerStarted","Data":"61ff5b8fb8f156db5633b47669156fbd4ef0ac3e3b9cd1cd5c188c4a7942ecc3"}
Oct 30 21:32:03.449627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.441839    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:03.457634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.454049    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.461627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.460354    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.481628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.476339    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5sl\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl\") pod \"e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") " pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.481628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.477179    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm" podStartSLOduration=3.477161695 podStartE2EDuration="3.477161695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:03.476650863 +0000 UTC m=+1306.399109162" watchObservedRunningTime="2026-10-30 21:32:03.477161695 +0000 UTC m=+1306.399620001"
Oct 30 21:32:03.548695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.543122    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx" podStartSLOduration=3.543098996 podStartE2EDuration="3.543098996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:03.541056961 +0000 UTC m=+1306.463515262" watchObservedRunningTime="2026-10-30 21:32:03.543098996 +0000 UTC m=+1306.465557338"
Oct 30 21:32:03.711692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.709506    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:03.725624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.722304    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:32:03.725624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.722364    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:32:03.978690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.976722    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:32:03.978690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:03.976783    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 21:32:04.305191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.305007    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"]
Oct 30 21:32:04.308774 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:04.308734    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbb82dc_8868_49ab_a6db_8a4fd4f62a08.slice/crio-d55b1a744849820ba0f1f52f1e265375b2c1603901b1e44bd9f57f48ececd843 WatchSource:0}: Error finding container d55b1a744849820ba0f1f52f1e265375b2c1603901b1e44bd9f57f48ececd843: Status 404 returned error can't find the container with id d55b1a744849820ba0f1f52f1e265375b2c1603901b1e44bd9f57f48ececd843
Oct 30 21:32:04.313241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.313195    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:04.313357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.313299    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:04.313357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.313344    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:04.454444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.454141    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" event={"ID":"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08","Type":"ContainerStarted","Data":"81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"}
Oct 30 21:32:04.454444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.454214    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" event={"ID":"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08","Type":"ContainerStarted","Data":"d55b1a744849820ba0f1f52f1e265375b2c1603901b1e44bd9f57f48ececd843"}
Oct 30 21:32:04.498190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.498121    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podStartSLOduration=2.49810122 podStartE2EDuration="2.49810122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:04.495169588 +0000 UTC m=+1307.417627891" watchObservedRunningTime="2026-10-30 21:32:04.49810122 +0000 UTC m=+1307.420559517"
Oct 30 21:32:04.714765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.710875    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:04.751079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.751002    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:32:04.753895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.753869    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:32:04.980492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.980405    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:32:04.980848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.980816    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:32:04.981586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:04.981565    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:32:05.741411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:05.741381    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:06.063836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.063795    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"]
Oct 30 21:32:06.082243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.082205    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"]
Oct 30 21:32:06.082432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.082333    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.099674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.099646    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c027ae\""
Oct 30 21:32:06.100071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.100026    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c027ae-dockercfg-hwn2d\""
Oct 30 21:32:06.218968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.218934    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxmw\" (UniqueName: \"kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw\") pod \"payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.219173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.219150    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.320309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.320224    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxmw\" (UniqueName: \"kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw\") pod \"payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.320467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.320353    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.321067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.321046    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.338950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.338912    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxmw\" (UniqueName: \"kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw\") pod \"payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.405500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.405469    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:06.496663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.482591    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:06.496663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.483998    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:06.720654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.720560    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"]
Oct 30 21:32:06.724298 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:06.724260    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419c5cea_64ae_44d8_b67c_624e030c0128.slice/crio-149c82a1a7c47be2c20bfeb8e704225c1303f8ddc2516970d7719968439893a1 WatchSource:0}: Error finding container 149c82a1a7c47be2c20bfeb8e704225c1303f8ddc2516970d7719968439893a1: Status 404 returned error can't find the container with id 149c82a1a7c47be2c20bfeb8e704225c1303f8ddc2516970d7719968439893a1
Oct 30 21:32:06.852258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.852230    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:32:06.931274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931174    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931234    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931291    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931354    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931402    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931471    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931536    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931596    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.931765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.931720    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp\") pod \"8a4cd3c8-25a0-477a-976c-b591bb067e68\" (UID: \"8a4cd3c8-25a0-477a-976c-b591bb067e68\") "
Oct 30 21:32:06.996276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.996236    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:06.996845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:06.996809    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:07.010668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.007187    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.010668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.009723    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.011338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.010952    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.011338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.011015    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.021085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.021043    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:07.034985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.034938    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.034985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.034981    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.035181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.034994    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.035181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.035007    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.035181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.035020    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.035181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.035032    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a4cd3c8-25a0-477a-976c-b591bb067e68-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.035181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.035045    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.038908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.038866    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "kube-api-access-d9lnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:07.042257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.041977    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token" pod "8a4cd3c8-25a0-477a-976c-b591bb067e68" (UID: "8a4cd3c8-25a0-477a-976c-b591bb067e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:07.136549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.136515    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.136549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.136542    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9lnp\" (UniqueName: \"kubernetes.io/projected/8a4cd3c8-25a0-477a-976c-b591bb067e68-kube-api-access-d9lnp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:07.518339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.518297    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6" event={"ID":"419c5cea-64ae-44d8-b67c-624e030c0128","Type":"ContainerStarted","Data":"149c82a1a7c47be2c20bfeb8e704225c1303f8ddc2516970d7719968439893a1"}
Oct 30 21:32:07.521547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.521514    6695 generic.go:356] "Generic (PLEG): container finished" podID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerID="f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489" exitCode=0
Oct 30 21:32:07.521732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.521578    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"
Oct 30 21:32:07.521732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.521632    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" event={"ID":"8a4cd3c8-25a0-477a-976c-b591bb067e68","Type":"ContainerDied","Data":"f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489"}
Oct 30 21:32:07.521732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.521700    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" event={"ID":"8a4cd3c8-25a0-477a-976c-b591bb067e68","Type":"ContainerDied","Data":"556913c1c1f5e8ad5b270b4a7992525a01dc84c3169c72cf44543fcb30491051"}
Oct 30 21:32:07.521732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.521732    6695 scope.go:122] "RemoveContainer" containerID="f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489"
Oct 30 21:32:07.536723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.536551    6695 scope.go:122] "RemoveContainer" containerID="55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"
Oct 30 21:32:07.554942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.554916    6695 scope.go:122] "RemoveContainer" containerID="f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489"
Oct 30 21:32:07.555356 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:07.555305    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489\": container with ID starting with f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489 not found: ID does not exist" containerID="f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489"
Oct 30 21:32:07.555473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.555349    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489"} err="failed to get container status \"f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489\": rpc error: code = NotFound desc = could not find container \"f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489\": container with ID starting with f1cb988f3324c0612ce9407e4232f4947e42f536752fa495849c861a1136a489 not found: ID does not exist"
Oct 30 21:32:07.555473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.555374    6695 scope.go:122] "RemoveContainer" containerID="55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"
Oct 30 21:32:07.555787 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:07.555731    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc\": container with ID starting with 55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc not found: ID does not exist" containerID="55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"
Oct 30 21:32:07.555787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.555775    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc"} err="failed to get container status \"55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc\": rpc error: code = NotFound desc = could not find container \"55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc\": container with ID starting with 55be96d8a5c76d0eb6be83c1fb57f36c2775720c9ef523298f6a0729463c26dc not found: ID does not exist"
Oct 30 21:32:07.576529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.576493    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"]
Oct 30 21:32:07.584989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.584952    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd"]
Oct 30 21:32:07.768434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.768388    6695 patch_prober.go:29] interesting pod/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:32:07.768640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:07.768476    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-342fc9-openshift-default-8947d46b9-bgvhd" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:08.535523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:08.535485    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6" event={"ID":"419c5cea-64ae-44d8-b67c-624e030c0128","Type":"ContainerStarted","Data":"ba4cfde60c7e1064be4ec74f1fd6e5ac0b28f3e645dd8ffda658fadcc9201528"}
Oct 30 21:32:08.536041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:08.535554    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:08.577843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:08.577736    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6" podStartSLOduration=1.977122708 podStartE2EDuration="3.577719921s" totalImagesPullingTime="1.600597213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:08.574652771 +0000 UTC m=+1311.497111069" watchObservedRunningTime="2026-10-30 21:32:08.577719921 +0000 UTC m=+1311.500178223"
Oct 30 21:32:09.059751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.059711    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"]
Oct 30 21:32:09.071739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.066769    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy"
Oct 30 21:32:09.072006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.071982    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy"
Oct 30 21:32:09.072477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.072401    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy"
Oct 30 21:32:09.072723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.072706    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy"
Oct 30 21:32:09.078290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.078265    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.162852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.162815    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"]
Oct 30 21:32:09.189627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.186948    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.189627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.187160    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdldj\" (UniqueName: \"kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj\") pod \"payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.189901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.189761    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a1ee3-dockercfg-g7hml\""
Oct 30 21:32:09.189901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.189836    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a1ee3\""
Oct 30 21:32:09.293662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.293625    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.293927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.293823    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdldj\" (UniqueName: \"kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj\") pod \"payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.295029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.295002    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.314698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.313902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdldj\" (UniqueName: \"kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj\") pod \"payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.314853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.314791    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" path="/var/lib/kubelet/pods/8a4cd3c8-25a0-477a-976c-b591bb067e68/volumes"
Oct 30 21:32:09.539824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.539789    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:09.851297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:09.851267    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"]
Oct 30 21:32:09.854415 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:09.854381    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17679a52_0f73_429f_93bb_681ef7a13b0f.slice/crio-8f03f21f82733255a91cbd54285e6dd2052f5e712ecaf862805308a0909829c4 WatchSource:0}: Error finding container 8f03f21f82733255a91cbd54285e6dd2052f5e712ecaf862805308a0909829c4: Status 404 returned error can't find the container with id 8f03f21f82733255a91cbd54285e6dd2052f5e712ecaf862805308a0909829c4
Oct 30 21:32:10.606987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:10.606949    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb" event={"ID":"17679a52-0f73-429f-93bb-681ef7a13b0f","Type":"ContainerStarted","Data":"d147681b4fbe4900f3a37cf93ba96f926197376ef4f86657146b9445d459692a"}
Oct 30 21:32:10.607496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:10.607011    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:10.607496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:10.607034    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb" event={"ID":"17679a52-0f73-429f-93bb-681ef7a13b0f","Type":"ContainerStarted","Data":"8f03f21f82733255a91cbd54285e6dd2052f5e712ecaf862805308a0909829c4"}
Oct 30 21:32:10.649554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:10.649496    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb" podStartSLOduration=2.649479536 podStartE2EDuration="2.649479536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:10.646139016 +0000 UTC m=+1313.568597314" watchObservedRunningTime="2026-10-30 21:32:10.649479536 +0000 UTC m=+1313.571937834"
Oct 30 21:32:14.256885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.256829    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"]
Oct 30 21:32:14.260031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.260002    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy"
Oct 30 21:32:14.260031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.260028    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4cd3c8-25a0-477a-976c-b591bb067e68" containerName="istio-proxy"
Oct 30 21:32:14.263170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.263149    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.278527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.278505    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7222e2-openshift-default-dockercfg-gz94k\""
Oct 30 21:32:14.294109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.294076    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"]
Oct 30 21:32:14.294949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.294917    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.295106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295015    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.295106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295078    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.299768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295142    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.299768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295210    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.299768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295273    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.299768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295403    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.299768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295532    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfwl\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.299768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.295666    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.408673    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.408769    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.408888    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.408999    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfwl\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.409057    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.409133    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.409173    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.409222    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.409281    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.409844    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.412628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.410098    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.427624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.423993    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.427624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.424660    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.427624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.424938    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.458496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.458265    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.471632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.467310    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfwl\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.471632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.469843    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.471632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.470283    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token\") pod \"e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") " pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:14.587624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:14.587184    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:15.102873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.102846    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"]
Oct 30 21:32:15.105376 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:15.105347    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e93ab7d_f44a_4035_b945_63ba14a87d0d.slice/crio-385f7a4c472d74ca12abb3bdf7d1c9c17a25f8e53dbc5e0a66423fdc6fa2521d WatchSource:0}: Error finding container 385f7a4c472d74ca12abb3bdf7d1c9c17a25f8e53dbc5e0a66423fdc6fa2521d: Status 404 returned error can't find the container with id 385f7a4c472d74ca12abb3bdf7d1c9c17a25f8e53dbc5e0a66423fdc6fa2521d
Oct 30 21:32:15.108231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.108200    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:15.108315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.108268    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:15.108315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.108300    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:15.469293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.469215    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:15.469293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.469284    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:15.754220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.754109    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" event={"ID":"8e93ab7d-f44a-4035-b945-63ba14a87d0d","Type":"ContainerStarted","Data":"0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"}
Oct 30 21:32:15.754525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.754273    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" event={"ID":"8e93ab7d-f44a-4035-b945-63ba14a87d0d","Type":"ContainerStarted","Data":"385f7a4c472d74ca12abb3bdf7d1c9c17a25f8e53dbc5e0a66423fdc6fa2521d"}
Oct 30 21:32:15.801387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:15.801330    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podStartSLOduration=1.801315361 podStartE2EDuration="1.801315361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:15.798383689 +0000 UTC m=+1318.720841986" watchObservedRunningTime="2026-10-30 21:32:15.801315361 +0000 UTC m=+1318.723773658"
Oct 30 21:32:16.588278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:16.588244    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:16.593805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:16.593784    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:16.762940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:16.762910    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:16.763915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:16.763892    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:32:18.249085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.249051    6695 scope.go:122] "RemoveContainer" containerID="809d1c5385d26fec5d10c21fb5f79aa0554b549b382ddcfad9c3f5713ffd99f3"
Oct 30 21:32:18.263915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.263889    6695 scope.go:122] "RemoveContainer" containerID="d067ac286572a2b6f22f329d5149596d04209afdba44a8d1e2f104ddd11e1df1"
Oct 30 21:32:18.276944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.276913    6695 scope.go:122] "RemoveContainer" containerID="82d2e9d444145c42f269cfbebda1e5c18fc62d447716021136fab5fd67683fbe"
Oct 30 21:32:18.287698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.287674    6695 scope.go:122] "RemoveContainer" containerID="12c1269bc4ce4393d547567773c42f06b432fdbd225645c9b52a07b5fed2f507"
Oct 30 21:32:18.298396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.298373    6695 scope.go:122] "RemoveContainer" containerID="a6c3a553db78ec27ab0d362a0b09560aacebb0e9fa7ead582a93106bfe6e01b6"
Oct 30 21:32:18.308057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.308031    6695 scope.go:122] "RemoveContainer" containerID="1e40aff0492317f3e8b8193650b763ca0eb16883690849d57ede48f4bc1a0487"
Oct 30 21:32:18.317255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.317233    6695 scope.go:122] "RemoveContainer" containerID="69a96583497f6684f2dc16df0c3cd9d0e57414ba6443b6c1ef6e097b223f6f36"
Oct 30 21:32:18.327523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.327494    6695 scope.go:122] "RemoveContainer" containerID="028cbfcb72c60e3fbacc394c4a9a6f980ac81b73a4899163fd44ae987cd47843"
Oct 30 21:32:18.336998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.336969    6695 scope.go:122] "RemoveContainer" containerID="e301c5d6ffc07ff52cc3624b6ee60dcc5f858ff88ea5ef8725e059b3747df675"
Oct 30 21:32:18.346252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.346228    6695 scope.go:122] "RemoveContainer" containerID="aa6fbfd346067de0ebac86051f90cbd1cf42572bc21e13c0cdf7225702e99dd6"
Oct 30 21:32:18.355922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.355897    6695 scope.go:122] "RemoveContainer" containerID="33121375c90b6290098973f81eab5687ca76e9c3d2155b4e34145f5b2c854cfd"
Oct 30 21:32:18.365740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:18.365717    6695 scope.go:122] "RemoveContainer" containerID="7b0e73ecd1387443fa1aa8b50c74cc97754ba38f32b6322ce3c88f163929856f"
Oct 30 21:32:19.547043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:19.547011    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:32:20.784227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:20.784196    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"]
Oct 30 21:32:20.796968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:20.796934    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:20.807697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:20.807661    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"]
Oct 30 21:32:20.815116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:20.815091    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7222e2-dockercfg-tm2sx\""
Oct 30 21:32:20.815267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:20.815121    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7222e2\""
Oct 30 21:32:21.087678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.084169    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.087678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.084515    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqff\" (UniqueName: \"kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.186256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.186216    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqff\" (UniqueName: \"kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.186429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.186362    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.188542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.187438    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.206227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.206196    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqff\" (UniqueName: \"kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.414273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.414187    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:21.650346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.649046    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:32:21.735302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:21.735277    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"]
Oct 30 21:32:21.738712 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:21.738677    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa73104_7772_43e0_a79d_85d7de79d72c.slice/crio-af9f2128d80078df65f07f6464f52aa7f9d13176ae59b6add634513c3ad7485a WatchSource:0}: Error finding container af9f2128d80078df65f07f6464f52aa7f9d13176ae59b6add634513c3ad7485a: Status 404 returned error can't find the container with id af9f2128d80078df65f07f6464f52aa7f9d13176ae59b6add634513c3ad7485a
Oct 30 21:32:22.157427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:22.157384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj" event={"ID":"bfa73104-7772-43e0-a79d-85d7de79d72c","Type":"ContainerStarted","Data":"117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595"}
Oct 30 21:32:22.157864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:22.157445    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj" event={"ID":"bfa73104-7772-43e0-a79d-85d7de79d72c","Type":"ContainerStarted","Data":"af9f2128d80078df65f07f6464f52aa7f9d13176ae59b6add634513c3ad7485a"}
Oct 30 21:32:22.157864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:22.157586    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:22.198800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:22.198749    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj" podStartSLOduration=2.198735552 podStartE2EDuration="2.198735552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:22.196557995 +0000 UTC m=+1325.119016294" watchObservedRunningTime="2026-10-30 21:32:22.198735552 +0000 UTC m=+1325.121193849"
Oct 30 21:32:25.947980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.947936    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"]
Oct 30 21:32:25.948665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.948172    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy" containerID="cri-o://84475e03713aee11cc4cae482f5aaee9ff39d298bc9876b2d78f6ea1a4702445" gracePeriod=30
Oct 30 21:32:25.951861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.951834    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"]
Oct 30 21:32:25.952135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.952053    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy" containerID="cri-o://f3c65fc847382165bc11b8a0777312557e36afcc8372e4c444216b74528d0bba" gracePeriod=30
Oct 30 21:32:25.966842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.966809    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"]
Oct 30 21:32:25.967262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.967230    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy" containerID="cri-o://7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da" gracePeriod=30
Oct 30 21:32:25.971415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.971392    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"]
Oct 30 21:32:25.971632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:25.971589    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy" containerID="cri-o://d266ef50fac4bd2b4f5ccc7f321425cde5d8fa04457b482a6f3b423a83fdd00e" gracePeriod=30
Oct 30 21:32:26.528124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.528086    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"]
Oct 30 21:32:26.534451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.534211    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.554636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.554571    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b1659a-openshift-default-dockercfg-l7xnq\""
Oct 30 21:32:26.555993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.555965    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"]
Oct 30 21:32:26.657532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657497    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pzd\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.657735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657540    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.657735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657568    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.657735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657696    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.657860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657745    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.657860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657842    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.657949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.657935    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.658024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.658013    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.658091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.658079    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.761757    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pzd\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.761819    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.761857    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.761897    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.761981    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.762057    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.762171    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.762273    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.762630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.762371    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.763249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.763228    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.764892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.763818    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.787711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.783873    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.787711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.784719    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.787711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.786104    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.814625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.810075    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pzd\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.818625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.815208    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.820189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.820121    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.825415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.825382    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo\") pod \"e2e-api-b-b1659a-openshift-default-594764768c-v58qh\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") " pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:26.880067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:26.880028    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:27.232414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:27.232313    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"]
Oct 30 21:32:27.236857 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:27.236824    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ba006c_a1ed_4618_81b6_91144a0c50a3.slice/crio-ba73a6df53122b1e2e767df23f5dbeb02e945f56731ce9438f3ec27b44a6a70c WatchSource:0}: Error finding container ba73a6df53122b1e2e767df23f5dbeb02e945f56731ce9438f3ec27b44a6a70c: Status 404 returned error can't find the container with id ba73a6df53122b1e2e767df23f5dbeb02e945f56731ce9438f3ec27b44a6a70c
Oct 30 21:32:27.239667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:27.239625    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:27.239773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:27.239726    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:27.239834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:27.239771    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:27.281651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:27.281596    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" event={"ID":"22ba006c-a1ed-4618-81b6-91144a0c50a3","Type":"ContainerStarted","Data":"ba73a6df53122b1e2e767df23f5dbeb02e945f56731ce9438f3ec27b44a6a70c"}
Oct 30 21:32:28.292476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:28.292409    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" event={"ID":"22ba006c-a1ed-4618-81b6-91144a0c50a3","Type":"ContainerStarted","Data":"953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"}
Oct 30 21:32:28.370964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:28.370912    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podStartSLOduration=2.370897255 podStartE2EDuration="2.370897255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:28.359532703 +0000 UTC m=+1331.281991000" watchObservedRunningTime="2026-10-30 21:32:28.370897255 +0000 UTC m=+1331.293355554"
Oct 30 21:32:28.881181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:28.881137    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:28.887151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:28.887117    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:29.194769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.194668    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"]
Oct 30 21:32:29.195054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.195024    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy" containerID="cri-o://75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635" gracePeriod=30
Oct 30 21:32:29.198995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.198971    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"]
Oct 30 21:32:29.199203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.199160    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy" containerID="cri-o://091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45" gracePeriod=30
Oct 30 21:32:29.219598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.219562    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"]
Oct 30 21:32:29.219800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.219658    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"]
Oct 30 21:32:29.219886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.219862    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy" containerID="cri-o://3e6d1ca18800c6c9a6e02a37546b4b5bdb955159a6f4b6e22d4df80918400304" gracePeriod=30
Oct 30 21:32:29.219966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.219888    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy" containerID="cri-o://7a60d8e66e897af2ad18b807c5298ce85fa953f5effce56fbf70a32a70b55e4b" gracePeriod=30
Oct 30 21:32:29.303026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.302989    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:29.303931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:29.303908    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:32:31.319487 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:31.319370    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:32:31.502681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.502554    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:32:31.524656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.524547    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:32:31.526837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.526811    6695 generic.go:356] "Generic (PLEG): container finished" podID="90e3522c-639c-4905-97be-e55c59e193f7" containerID="f3c65fc847382165bc11b8a0777312557e36afcc8372e4c444216b74528d0bba" exitCode=0
Oct 30 21:32:31.526980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.526856    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"
Oct 30 21:32:31.526980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.526906    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm" event={"ID":"90e3522c-639c-4905-97be-e55c59e193f7","Type":"ContainerDied","Data":"f3c65fc847382165bc11b8a0777312557e36afcc8372e4c444216b74528d0bba"}
Oct 30 21:32:31.527066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.526983    6695 scope.go:122] "RemoveContainer" containerID="f3c65fc847382165bc11b8a0777312557e36afcc8372e4c444216b74528d0bba"
Oct 30 21:32:31.529181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.529157    6695 generic.go:356] "Generic (PLEG): container finished" podID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerID="84475e03713aee11cc4cae482f5aaee9ff39d298bc9876b2d78f6ea1a4702445" exitCode=0
Oct 30 21:32:31.529477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.529352    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" event={"ID":"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6","Type":"ContainerDied","Data":"84475e03713aee11cc4cae482f5aaee9ff39d298bc9876b2d78f6ea1a4702445"}
Oct 30 21:32:31.537146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.537118    6695 generic.go:356] "Generic (PLEG): container finished" podID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerID="d266ef50fac4bd2b4f5ccc7f321425cde5d8fa04457b482a6f3b423a83fdd00e" exitCode=0
Oct 30 21:32:31.537343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.537158    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" event={"ID":"0afb3395-41cd-4c84-89cb-22e23b9d08d3","Type":"ContainerDied","Data":"d266ef50fac4bd2b4f5ccc7f321425cde5d8fa04457b482a6f3b423a83fdd00e"}
Oct 30 21:32:31.542094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.542070    6695 generic.go:356] "Generic (PLEG): container finished" podID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerID="7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da" exitCode=0
Oct 30 21:32:31.542929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.542909    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"
Oct 30 21:32:31.543068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.543050    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" event={"ID":"cac9bb90-1551-4066-ad6c-df13b6e99042","Type":"ContainerDied","Data":"7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da"}
Oct 30 21:32:31.543134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.543092    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm" event={"ID":"cac9bb90-1551-4066-ad6c-df13b6e99042","Type":"ContainerDied","Data":"2736446f3ad4b436f6f26f9f78a6bc7301a2d034a85e38a6b50fb9f21ae6ee3f"}
Oct 30 21:32:31.548172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.548147    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:32:31.548539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.548522    6695 scope.go:122] "RemoveContainer" containerID="7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da"
Oct 30 21:32:31.564079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.564045    6695 scope.go:122] "RemoveContainer" containerID="7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da"
Oct 30 21:32:31.565338 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:31.565300    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da\": container with ID starting with 7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da not found: ID does not exist" containerID="7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da"
Oct 30 21:32:31.565450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.565378    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da"} err="failed to get container status \"7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da\": rpc error: code = NotFound desc = could not find container \"7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da\": container with ID starting with 7c9c1c3f1e9fdf125bd0e2cfe69226e39e520fe9af5b1e7ffc5112fe0675e3da not found: ID does not exist"
Oct 30 21:32:31.568360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.568332    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:32:31.573914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.573886    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574090    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.574198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574141    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.574198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574190    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574263    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.574366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574304    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574349    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.574467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574386    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.574467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574420    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.574467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574460    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.574636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574512    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.574636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574546    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574656    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574701    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.574750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574713    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.574750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574736    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.574946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574808    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574855    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.574946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574893    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.574946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574924    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.574946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.574937    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.575187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575001    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.575187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575059    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.575187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575101    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.575187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575176    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.575395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575222    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.575395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575349    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket\") pod \"cac9bb90-1551-4066-ad6c-df13b6e99042\" (UID: \"cac9bb90-1551-4066-ad6c-df13b6e99042\") "
Oct 30 21:32:31.575492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575460    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo\") pod \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\" (UID: \"0afb3395-41cd-4c84-89cb-22e23b9d08d3\") "
Oct 30 21:32:31.575547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575489    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data\") pod \"90e3522c-639c-4905-97be-e55c59e193f7\" (UID: \"90e3522c-639c-4905-97be-e55c59e193f7\") "
Oct 30 21:32:31.576780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.575964    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:31.576780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.576506    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cac9bb90-1551-4066-ad6c-df13b6e99042-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.576780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.576522    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.576780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.576531    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.577192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.576882    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.577192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.576898    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.577192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.577113    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.577361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.577338    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.577418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.577355    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.577680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.577646    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:31.577916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.577859    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.578020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.577913    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.578471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.578447    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.581895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.581586    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:31.582100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.582073    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:31.583413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.583388    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "kube-api-access-tl779". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.583783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.583761    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.583926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.583790    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.584030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.583885    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.584127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.584088    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.584127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.584106    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:31.584387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.584361    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "kube-api-access-556gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.584460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.584426    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.586095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.586061    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.586095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.586084    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.587151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.587122    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token" pod "0afb3395-41cd-4c84-89cb-22e23b9d08d3" (UID: "0afb3395-41cd-4c84-89cb-22e23b9d08d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.590135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.590110    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549" pod "90e3522c-639c-4905-97be-e55c59e193f7" (UID: "90e3522c-639c-4905-97be-e55c59e193f7"). InnerVolumeSpecName "kube-api-access-r6549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.592207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.592183    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo" pod "cac9bb90-1551-4066-ad6c-df13b6e99042" (UID: "cac9bb90-1551-4066-ad6c-df13b6e99042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:31.683021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.682985    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683055    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683125    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683189    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683285    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683584    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683640    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683677    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.683739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.683717    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo\") pod \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\" (UID: \"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6\") "
Oct 30 21:32:31.684510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684210    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.684760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684510    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684841    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684864    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684892    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e3522c-639c-4905-97be-e55c59e193f7-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684908    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl779\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-kube-api-access-tl779\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684923    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684937    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684950    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684962    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684974    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684985    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.684997    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556gq\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-kube-api-access-556gq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685009    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685022    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685035    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685048    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685061    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685072    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685084    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685097    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685113    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.685167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685093    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.686385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.685701    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.686583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.686468    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afb3395-41cd-4c84-89cb-22e23b9d08d3-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.686583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.686498    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6549\" (UniqueName: \"kubernetes.io/projected/90e3522c-639c-4905-97be-e55c59e193f7-kube-api-access-r6549\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.686583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.686514    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e3522c-639c-4905-97be-e55c59e193f7-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.686583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.686530    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e3522c-639c-4905-97be-e55c59e193f7-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.686583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.686545    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cac9bb90-1551-4066-ad6c-df13b6e99042-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.686583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.686562    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cac9bb90-1551-4066-ad6c-df13b6e99042-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.687050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.687025    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.687247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.687226    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:31.687330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.687314    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:31.688719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.688684    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76" pod "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" (UID: "ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6"). InnerVolumeSpecName "kube-api-access-9sq76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:31.787851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787818    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.787851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787847    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sq76\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-kube-api-access-9sq76\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.787851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787856    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.788119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787865    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.788119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787874    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.788119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787882    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.788119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787891    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.788119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.787899    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:31.931113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.931073    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"]
Oct 30 21:32:31.934803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.934770    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbfnm"]
Oct 30 21:32:31.964455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.964419    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"]
Oct 30 21:32:31.969341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:31.969311    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zjkvm"]
Oct 30 21:32:32.554281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.554251    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"
Oct 30 21:32:32.554754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.554255    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps" event={"ID":"0afb3395-41cd-4c84-89cb-22e23b9d08d3","Type":"ContainerDied","Data":"6ef13f3abdb8434a798e16cbb5a5b4aa08a39d397901c7c513b87b7faedc2ae2"}
Oct 30 21:32:32.554754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.554402    6695 scope.go:122] "RemoveContainer" containerID="d266ef50fac4bd2b4f5ccc7f321425cde5d8fa04457b482a6f3b423a83fdd00e"
Oct 30 21:32:32.558778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.558755    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"
Oct 30 21:32:32.558924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.558776    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr" event={"ID":"ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6","Type":"ContainerDied","Data":"03440bcb287876623a5c87efb45c6240ff97a72760030102769ddf073b0f5878"}
Oct 30 21:32:32.566973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.566944    6695 scope.go:122] "RemoveContainer" containerID="84475e03713aee11cc4cae482f5aaee9ff39d298bc9876b2d78f6ea1a4702445"
Oct 30 21:32:32.610303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.610270    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"]
Oct 30 21:32:32.620810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.620775    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-pskps"]
Oct 30 21:32:32.647091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.647052    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"]
Oct 30 21:32:32.658365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.658326    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-m7drr"]
Oct 30 21:32:32.840722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.840628    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"]
Oct 30 21:32:32.841349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841327    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841350    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841373    6695 cpu_manager.go:414] "containerMap: removing container" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841380    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841422    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841429    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841440    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy"
Oct 30 21:32:32.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841447    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy"
Oct 30 21:32:32.841839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841634    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e3522c-639c-4905-97be-e55c59e193f7" containerName="istio-proxy"
Oct 30 21:32:32.841839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841658    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" containerName="istio-proxy"
Oct 30 21:32:32.841839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841668    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" containerName="istio-proxy"
Oct 30 21:32:32.841839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.841686    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" containerName="istio-proxy"
Oct 30 21:32:32.846806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.846784    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:32.862918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.862892    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"]
Oct 30 21:32:32.865052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.865018    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b1659a\""
Oct 30 21:32:32.930815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.930777    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"]
Oct 30 21:32:32.957629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.941298    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ssj\" (UniqueName: \"kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj\") pod \"payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:32.957629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.941425    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:32.957629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:32.941902    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b1659a-dockercfg-25pg8\""
Oct 30 21:32:33.074627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.073992    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ssj\" (UniqueName: \"kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj\") pod \"payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:33.074627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.074119    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:33.091629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.075093    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:33.131677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.104668    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"]
Oct 30 21:32:33.131677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.104799    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.208637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.204831    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ssj\" (UniqueName: \"kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj\") pod \"payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:33.216705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.212811    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5z98\" (UniqueName: \"kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98\") pod \"payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.216705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.213005    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.238426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.238384    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0afb3395-41cd-4c84-89cb-22e23b9d08d3" path="/var/lib/kubelet/pods/0afb3395-41cd-4c84-89cb-22e23b9d08d3/volumes"
Oct 30 21:32:33.239222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.239196    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e3522c-639c-4905-97be-e55c59e193f7" path="/var/lib/kubelet/pods/90e3522c-639c-4905-97be-e55c59e193f7/volumes"
Oct 30 21:32:33.239911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.239892    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6" path="/var/lib/kubelet/pods/ac6a72a5-8ccb-4ec9-9d3b-bff162c10bf6/volumes"
Oct 30 21:32:33.240960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.240939    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac9bb90-1551-4066-ad6c-df13b6e99042" path="/var/lib/kubelet/pods/cac9bb90-1551-4066-ad6c-df13b6e99042/volumes"
Oct 30 21:32:33.241414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.241398    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:32:33.314884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.314844    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5z98\" (UniqueName: \"kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98\") pod \"payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.315059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.315027    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.315956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.315927    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.337965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.337926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5z98\" (UniqueName: \"kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98\") pod \"payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.460202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.460115    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:33.465126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.465095    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:33.799019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.798960    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"]
Oct 30 21:32:33.801763 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:33.801735    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62a5942_dd5c_4c93_891f_61f50f849431.slice/crio-47d03cf5bfd8ad3c70011c79b00310c5ca64cac1cb792c1d2388276b4399296c WatchSource:0}: Error finding container 47d03cf5bfd8ad3c70011c79b00310c5ca64cac1cb792c1d2388276b4399296c: Status 404 returned error can't find the container with id 47d03cf5bfd8ad3c70011c79b00310c5ca64cac1cb792c1d2388276b4399296c
Oct 30 21:32:33.816664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:33.816634    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"]
Oct 30 21:32:33.818138 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:32:33.818100    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337aed11_fa01_4112_aba4_a621ad0bc85b.slice/crio-3294a961f84977cfc9610574ce7b4c0bd73f3230090d8fe2e1e73ed126b21476 WatchSource:0}: Error finding container 3294a961f84977cfc9610574ce7b4c0bd73f3230090d8fe2e1e73ed126b21476: Status 404 returned error can't find the container with id 3294a961f84977cfc9610574ce7b4c0bd73f3230090d8fe2e1e73ed126b21476
Oct 30 21:32:34.495711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.495677    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:32:34.536331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.536265    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:32:34.593095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.593063    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:32:34.593316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.593089    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr" event={"ID":"337aed11-fa01-4112-aba4-a621ad0bc85b","Type":"ContainerStarted","Data":"f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210"}
Oct 30 21:32:34.593316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.593178    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr" event={"ID":"337aed11-fa01-4112-aba4-a621ad0bc85b","Type":"ContainerStarted","Data":"3294a961f84977cfc9610574ce7b4c0bd73f3230090d8fe2e1e73ed126b21476"}
Oct 30 21:32:34.594686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.594660    6695 generic.go:356] "Generic (PLEG): container finished" podID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerID="75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635" exitCode=0
Oct 30 21:32:34.594819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.594694    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"
Oct 30 21:32:34.594819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.594785    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" event={"ID":"0b64f18b-9634-486a-9ec2-a52afa35fd99","Type":"ContainerDied","Data":"75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635"}
Oct 30 21:32:34.594959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.594828    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv" event={"ID":"0b64f18b-9634-486a-9ec2-a52afa35fd99","Type":"ContainerDied","Data":"1a111b34f2c17d5a1fc26b2945fce53f27d983418757d1406e650f3a76e25932"}
Oct 30 21:32:34.594959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.594860    6695 scope.go:122] "RemoveContainer" containerID="75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635"
Oct 30 21:32:34.597086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.597061    6695 generic.go:356] "Generic (PLEG): container finished" podID="fae9c26d-4245-4118-a693-8315be8781b9" containerID="3e6d1ca18800c6c9a6e02a37546b4b5bdb955159a6f4b6e22d4df80918400304" exitCode=0
Oct 30 21:32:34.597247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.597145    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c" event={"ID":"fae9c26d-4245-4118-a693-8315be8781b9","Type":"ContainerDied","Data":"3e6d1ca18800c6c9a6e02a37546b4b5bdb955159a6f4b6e22d4df80918400304"}
Oct 30 21:32:34.597247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.597172    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"
Oct 30 21:32:34.599472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.599448    6695 generic.go:356] "Generic (PLEG): container finished" podID="33268e64-f853-4363-9bbf-191ff48e9205" containerID="7a60d8e66e897af2ad18b807c5298ce85fa953f5effce56fbf70a32a70b55e4b" exitCode=0
Oct 30 21:32:34.599655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.599525    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" event={"ID":"33268e64-f853-4363-9bbf-191ff48e9205","Type":"ContainerDied","Data":"7a60d8e66e897af2ad18b807c5298ce85fa953f5effce56fbf70a32a70b55e4b"}
Oct 30 21:32:34.601747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.601726    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"
Oct 30 21:32:34.602039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.601967    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" event={"ID":"7b69f28c-173a-497d-84c5-d07cce3a16fb","Type":"ContainerDied","Data":"091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45"}
Oct 30 21:32:34.602439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.601598    6695 generic.go:356] "Generic (PLEG): container finished" podID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerID="091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45" exitCode=0
Oct 30 21:32:34.602666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.602646    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd" event={"ID":"7b69f28c-173a-497d-84c5-d07cce3a16fb","Type":"ContainerDied","Data":"3a24fad80078198a39b9f72e1dd70fc19d946ee9228042286dcd23f9661b5f2e"}
Oct 30 21:32:34.609377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.609203    6695 scope.go:122] "RemoveContainer" containerID="75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635"
Oct 30 21:32:34.609874 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:34.609759    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635\": container with ID starting with 75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635 not found: ID does not exist" containerID="75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635"
Oct 30 21:32:34.609874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.609793    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635"} err="failed to get container status \"75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635\": rpc error: code = NotFound desc = could not find container \"75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635\": container with ID starting with 75bd8727c277c89ac17fd351ddbffddabf6a5264ff1d465aaa661ba0fb4d3635 not found: ID does not exist"
Oct 30 21:32:34.609874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.609816    6695 scope.go:122] "RemoveContainer" containerID="3e6d1ca18800c6c9a6e02a37546b4b5bdb955159a6f4b6e22d4df80918400304"
Oct 30 21:32:34.610454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.610427    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm" event={"ID":"a62a5942-dd5c-4c93-891f-61f50f849431","Type":"ContainerStarted","Data":"4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f"}
Oct 30 21:32:34.610545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.610478    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm" event={"ID":"a62a5942-dd5c-4c93-891f-61f50f849431","Type":"ContainerStarted","Data":"47d03cf5bfd8ad3c70011c79b00310c5ca64cac1cb792c1d2388276b4399296c"}
Oct 30 21:32:34.611407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.611388    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:34.611655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.611638    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:32:34.623343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.623323    6695 scope.go:122] "RemoveContainer" containerID="091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45"
Oct 30 21:32:34.634663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634634    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.634784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634683    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.634784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634712    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.634784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634730    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.634784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634776    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.634999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634809    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.634999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634874    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.634999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634912    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.634999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634943    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.634999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634966    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.634999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.634987    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.635280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635059    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.635280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635184    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.635280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635233    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy\") pod \"7b69f28c-173a-497d-84c5-d07cce3a16fb\" (UID: \"7b69f28c-173a-497d-84c5-d07cce3a16fb\") "
Oct 30 21:32:34.635428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635323    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.635428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635389    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.635428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635423    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.635579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635489    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo\") pod \"0b64f18b-9634-486a-9ec2-a52afa35fd99\" (UID: \"0b64f18b-9634-486a-9ec2-a52afa35fd99\") "
Oct 30 21:32:34.635695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635293    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.635762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635374    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.635762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635702    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.635762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635574    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.635762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635413    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.636039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.635987    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.636098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636034    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:34.636098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636034    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.636300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636281    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:34.636420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636397    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.636661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636634    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b69f28c-173a-497d-84c5-d07cce3a16fb-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636670    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636686    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636699    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636712    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b64f18b-9634-486a-9ec2-a52afa35fd99-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636726    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636739    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.636755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636751    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.637174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.636766    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.638958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.638907    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.638958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.638911    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:34.639091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.639017    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.639149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.639094    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "kube-api-access-q44m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.639149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.639143    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy" pod "7b69f28c-173a-497d-84c5-d07cce3a16fb" (UID: "7b69f28c-173a-497d-84c5-d07cce3a16fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.639257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.638681    6695 scope.go:122] "RemoveContainer" containerID="091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45"
Oct 30 21:32:34.639638 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:34.639527    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45\": container with ID starting with 091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45 not found: ID does not exist" containerID="091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45"
Oct 30 21:32:34.639638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.639565    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45"} err="failed to get container status \"091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45\": rpc error: code = NotFound desc = could not find container \"091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45\": container with ID starting with 091072f95120811f54654a2be5eeb56e9969e3b2b2c47058a58d814dd916dd45 not found: ID does not exist"
Oct 30 21:32:34.639998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.639969    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.640257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.640236    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:34.640473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.640450    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn" pod "0b64f18b-9634-486a-9ec2-a52afa35fd99" (UID: "0b64f18b-9634-486a-9ec2-a52afa35fd99"). InnerVolumeSpecName "kube-api-access-6dwnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.716452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.716394    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr" podStartSLOduration=2.716377455 podStartE2EDuration="2.716377455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:34.714716647 +0000 UTC m=+1337.637174944" watchObservedRunningTime="2026-10-30 21:32:34.716377455 +0000 UTC m=+1337.638835754"
Oct 30 21:32:34.737497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737422    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.737497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737464    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.737497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737484    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737522    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737562    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737579    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737663    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737707    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737768    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737765    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.737819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737808    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737859    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737888    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737893    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737898    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.737983    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738059    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738086    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738140    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738170    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738199    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738241    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data\") pod \"33268e64-f853-4363-9bbf-191ff48e9205\" (UID: \"33268e64-f853-4363-9bbf-191ff48e9205\") "
Oct 30 21:32:34.738303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738301    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.738933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738365    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket\") pod \"fae9c26d-4245-4118-a693-8315be8781b9\" (UID: \"fae9c26d-4245-4118-a693-8315be8781b9\") "
Oct 30 21:32:34.738933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.738714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:34.739060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739043    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.739231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739211    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.739389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739238    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739395    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwnn\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-kube-api-access-6dwnn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739414    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739431    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44m2\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-kube-api-access-q44m2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739446    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739460    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b64f18b-9634-486a-9ec2-a52afa35fd99-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739473    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739486    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739499    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.739517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739514    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739731    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.740258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.739894    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.740482    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.740508    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.740522    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33268e64-f853-4363-9bbf-191ff48e9205-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.740537    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.740552    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b69f28c-173a-497d-84c5-d07cce3a16fb-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.740723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.740564    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae9c26d-4245-4118-a693-8315be8781b9-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.741509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.741487    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:34.741765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.741687    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "kube-api-access-5ddn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.741765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.741697    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:34.741765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.741738    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.741765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.741743    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.741765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.741758    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.742140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.742125    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token" pod "fae9c26d-4245-4118-a693-8315be8781b9" (UID: "fae9c26d-4245-4118-a693-8315be8781b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.742980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.742962    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg" pod "33268e64-f853-4363-9bbf-191ff48e9205" (UID: "33268e64-f853-4363-9bbf-191ff48e9205"). InnerVolumeSpecName "kube-api-access-5mgbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.790851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.790784    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm" podStartSLOduration=2.790765115 podStartE2EDuration="2.790765115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:34.78922496 +0000 UTC m=+1337.711683270" watchObservedRunningTime="2026-10-30 21:32:34.790765115 +0000 UTC m=+1337.713223414"
Oct 30 21:32:34.841902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841867    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mgbg\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-kube-api-access-5mgbg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.841902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841898    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33268e64-f853-4363-9bbf-191ff48e9205-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.841902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841908    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841915    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841924    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33268e64-f853-4363-9bbf-191ff48e9205-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841932    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae9c26d-4245-4118-a693-8315be8781b9-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841941    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841951    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33268e64-f853-4363-9bbf-191ff48e9205-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841964    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddn8\" (UniqueName: \"kubernetes.io/projected/fae9c26d-4245-4118-a693-8315be8781b9-kube-api-access-5ddn8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:34.842351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:34.841973    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae9c26d-4245-4118-a693-8315be8781b9-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:35.025658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.025108    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"]
Oct 30 21:32:35.029005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.028897    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xh6nv"]
Oct 30 21:32:35.057756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.057723    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"]
Oct 30 21:32:35.061145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.061115    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-m6t2c"]
Oct 30 21:32:35.097680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.097587    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"]
Oct 30 21:32:35.101367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.101333    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-4mnjd"]
Oct 30 21:32:35.217207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.217172    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" path="/var/lib/kubelet/pods/0b64f18b-9634-486a-9ec2-a52afa35fd99/volumes"
Oct 30 21:32:35.217970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.217943    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" path="/var/lib/kubelet/pods/7b69f28c-173a-497d-84c5-d07cce3a16fb/volumes"
Oct 30 21:32:35.218636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.218598    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae9c26d-4245-4118-a693-8315be8781b9" path="/var/lib/kubelet/pods/fae9c26d-4245-4118-a693-8315be8781b9/volumes"
Oct 30 21:32:35.631479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.631435    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj" event={"ID":"33268e64-f853-4363-9bbf-191ff48e9205","Type":"ContainerDied","Data":"6a33d1833fcabe2ec7ffa92a533eb816f00c7e7e388ab614dc12caf6d7e411fa"}
Oct 30 21:32:35.631753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.631482    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"
Oct 30 21:32:35.631921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.631763    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:35.631921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.631801    6695 scope.go:122] "RemoveContainer" containerID="7a60d8e66e897af2ad18b807c5298ce85fa953f5effce56fbf70a32a70b55e4b"
Oct 30 21:32:35.685592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.685557    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"]
Oct 30 21:32:35.693967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:35.693927    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-st4hj"]
Oct 30 21:32:37.215896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:37.215866    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33268e64-f853-4363-9bbf-191ff48e9205" path="/var/lib/kubelet/pods/33268e64-f853-4363-9bbf-191ff48e9205/volumes"
Oct 30 21:32:44.175522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:44.175488    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"]
Oct 30 21:32:44.207821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:44.207783    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"]
Oct 30 21:32:46.715627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:46.713360    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:32:46.715627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:46.713956    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:32:48.032142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:48.032100    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"]
Oct 30 21:32:48.049476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:48.049438    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39cf64a-s8t4q"]
Oct 30 21:32:49.176883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.176747    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx" podUID="3f853e10-748b-4fca-acab-bd324a88b1f1" containerName="payload-processing" containerID="cri-o://10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc" gracePeriod=25
Oct 30 21:32:49.208145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.208097    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm" podUID="ff3c3028-b40a-4e77-8826-59b7efe0f6e4" containerName="payload-pre-processing" containerID="cri-o://820d4982ed2dd68dd253183beaba3187637b7f349e771d6ecd3cdd732615173e" gracePeriod=25
Oct 30 21:32:49.276765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.276733    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580" path="/var/lib/kubelet/pods/fc4718f6-dc6f-4e9a-a6ea-2dcdcac68580/volumes"
Oct 30 21:32:49.825629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.817034    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:49.867627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.865559    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj\" (UniqueName: \"kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj\") pod \"3f853e10-748b-4fca-acab-bd324a88b1f1\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") "
Oct 30 21:32:49.867627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.865789    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume\") pod \"3f853e10-748b-4fca-acab-bd324a88b1f1\" (UID: \"3f853e10-748b-4fca-acab-bd324a88b1f1\") "
Oct 30 21:32:49.879901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.879863    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume" pod "3f853e10-748b-4fca-acab-bd324a88b1f1" (UID: "3f853e10-748b-4fca-acab-bd324a88b1f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:49.880509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.880427    6695 generic.go:356] "Generic (PLEG): container finished" podID="ff3c3028-b40a-4e77-8826-59b7efe0f6e4" containerID="820d4982ed2dd68dd253183beaba3187637b7f349e771d6ecd3cdd732615173e" exitCode=0
Oct 30 21:32:49.880694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.880669    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm" event={"ID":"ff3c3028-b40a-4e77-8826-59b7efe0f6e4","Type":"ContainerDied","Data":"820d4982ed2dd68dd253183beaba3187637b7f349e771d6ecd3cdd732615173e"}
Oct 30 21:32:49.883835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.883809    6695 generic.go:356] "Generic (PLEG): container finished" podID="3f853e10-748b-4fca-acab-bd324a88b1f1" containerID="10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc" exitCode=0
Oct 30 21:32:49.883988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.883969    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx" event={"ID":"3f853e10-748b-4fca-acab-bd324a88b1f1","Type":"ContainerDied","Data":"10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc"}
Oct 30 21:32:49.884064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.884022    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx" event={"ID":"3f853e10-748b-4fca-acab-bd324a88b1f1","Type":"ContainerDied","Data":"61ff5b8fb8f156db5633b47669156fbd4ef0ac3e3b9cd1cd5c188c4a7942ecc3"}
Oct 30 21:32:49.884120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.884055    6695 scope.go:122] "RemoveContainer" containerID="10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc"
Oct 30 21:32:49.884189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.884161    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj" pod "3f853e10-748b-4fca-acab-bd324a88b1f1" (UID: "3f853e10-748b-4fca-acab-bd324a88b1f1"). InnerVolumeSpecName "kube-api-access-8crwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:49.884244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.884236    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"
Oct 30 21:32:49.906402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.906381    6695 scope.go:122] "RemoveContainer" containerID="10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc"
Oct 30 21:32:49.906866 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:32:49.906834    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc\": container with ID starting with 10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc not found: ID does not exist" containerID="10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc"
Oct 30 21:32:49.907012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.906992    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc"} err="failed to get container status \"10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc\": rpc error: code = NotFound desc = could not find container \"10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc\": container with ID starting with 10a3ba5a8a390d65b167af20aff870c0cbf3922b067ed129ab7c95fd6d38a0bc not found: ID does not exist"
Oct 30 21:32:49.916847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.916811    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:49.971106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.971068    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f853e10-748b-4fca-acab-bd324a88b1f1-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:49.971106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.971104    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8crwj\" (UniqueName: \"kubernetes.io/projected/3f853e10-748b-4fca-acab-bd324a88b1f1-kube-api-access-8crwj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:49.975878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.975817    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"]
Oct 30 21:32:49.980967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:49.980937    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-7564f49867-mldcx"]
Oct 30 21:32:50.071767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.071687    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume\") pod \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") "
Oct 30 21:32:50.072117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.072093    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx\" (UniqueName: \"kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx\") pod \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\" (UID: \"ff3c3028-b40a-4e77-8826-59b7efe0f6e4\") "
Oct 30 21:32:50.072237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.072157    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume" pod "ff3c3028-b40a-4e77-8826-59b7efe0f6e4" (UID: "ff3c3028-b40a-4e77-8826-59b7efe0f6e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:50.072621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.072581    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:50.074409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.074378    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx" pod "ff3c3028-b40a-4e77-8826-59b7efe0f6e4" (UID: "ff3c3028-b40a-4e77-8826-59b7efe0f6e4"). InnerVolumeSpecName "kube-api-access-7vqhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:50.176627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.175243    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqhx\" (UniqueName: \"kubernetes.io/projected/ff3c3028-b40a-4e77-8826-59b7efe0f6e4-kube-api-access-7vqhx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:32:50.899470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.899299    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm" event={"ID":"ff3c3028-b40a-4e77-8826-59b7efe0f6e4","Type":"ContainerDied","Data":"8daf1f8ef19fcf0ff82cc9cbe49fdf7752abf6fab67d935ddf3cc589a670d88f"}
Oct 30 21:32:50.899470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.899407    6695 scope.go:122] "RemoveContainer" containerID="820d4982ed2dd68dd253183beaba3187637b7f349e771d6ecd3cdd732615173e"
Oct 30 21:32:50.900042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:50.899527    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"
Oct 30 21:32:51.049633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:51.024226    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"]
Oct 30 21:32:51.081115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:51.081037    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-586bdd7995-tw4qm"]
Oct 30 21:32:51.259920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:51.259836    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f853e10-748b-4fca-acab-bd324a88b1f1" path="/var/lib/kubelet/pods/3f853e10-748b-4fca-acab-bd324a88b1f1/volumes"
Oct 30 21:32:51.260833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:51.260800    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3c3028-b40a-4e77-8826-59b7efe0f6e4" path="/var/lib/kubelet/pods/ff3c3028-b40a-4e77-8826-59b7efe0f6e4/volumes"
Oct 30 21:32:57.590169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:57.590118    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": read tcp 10.134.0.2:43842->10.134.0.150:15021: read: connection reset by peer" start-of-body=
Oct 30 21:32:57.590639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:57.590194    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": read tcp 10.134.0.2:43842->10.134.0.150:15021: read: connection reset by peer"
Oct 30 21:32:58.092702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.092670    6695 generic.go:356] "Generic (PLEG): container finished" podID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerID="3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9" exitCode=0
Oct 30 21:32:58.092900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.092747    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" event={"ID":"6259c498-a1b5-4aba-8d29-1fe6a590d24f","Type":"ContainerDied","Data":"3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"}
Oct 30 21:32:58.093234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.093213    6695 scope.go:122] "RemoveContainer" containerID="3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"
Oct 30 21:32:58.094660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.094640    6695 generic.go:356] "Generic (PLEG): container finished" podID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerID="81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb" exitCode=0
Oct 30 21:32:58.094827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.094815    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" event={"ID":"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08","Type":"ContainerDied","Data":"81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"}
Oct 30 21:32:58.094875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.094851    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:58.094936 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.094922    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:58.094979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.094959    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:58.095108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.095096    6695 scope.go:122] "RemoveContainer" containerID="81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"
Oct 30 21:32:58.096388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.096364    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:58.096463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.096427    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:58.096463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.096459    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:58.857612 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:58.857551    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"]
Oct 30 21:32:59.198710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.196418    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" event={"ID":"6259c498-a1b5-4aba-8d29-1fe6a590d24f","Type":"ContainerStarted","Data":"b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d"}
Oct 30 21:32:59.206625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.205992    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" event={"ID":"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08","Type":"ContainerStarted","Data":"06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075"}
Oct 30 21:32:59.716629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.713187    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:32:59.716629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.713314    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 21:32:59.716629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.713370    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 21:32:59.761631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.757247    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:32:59.761631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.757382    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:32:59.761631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:32:59.757422    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:33:00.722630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:00.721039    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 21:33:00.722630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:00.721104    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 21:33:00.757628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:00.757049    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:33:00.757628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:00.757109    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:33:01.320545 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:33:01.320505    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:33:01.737534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:01.737433    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 21:33:01.738172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:01.738124    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 21:33:01.762695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:01.762650    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:33:01.764539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:01.764488    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:33:02.718073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:02.718029    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 21:33:02.718397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:02.718364    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 21:33:02.756071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:02.756026    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:33:02.763783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:02.763727    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:33:03.715062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:03.715022    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 21:33:03.715363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:03.715331    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 21:33:03.757624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:03.756050    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:33:03.757624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:03.756124    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:33:03.861758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:03.861676    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb" podUID="17679a52-0f73-429f-93bb-681ef7a13b0f" containerName="payload-processing" containerID="cri-o://d147681b4fbe4900f3a37cf93ba96f926197376ef4f86657146b9445d459692a" gracePeriod=25
Oct 30 21:33:04.401345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.401320    6695 generic.go:356] "Generic (PLEG): container finished" podID="17679a52-0f73-429f-93bb-681ef7a13b0f" containerID="d147681b4fbe4900f3a37cf93ba96f926197376ef4f86657146b9445d459692a" exitCode=0
Oct 30 21:33:04.401490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.401390    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb" event={"ID":"17679a52-0f73-429f-93bb-681ef7a13b0f","Type":"ContainerDied","Data":"d147681b4fbe4900f3a37cf93ba96f926197376ef4f86657146b9445d459692a"}
Oct 30 21:33:04.401490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.401470    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb" event={"ID":"17679a52-0f73-429f-93bb-681ef7a13b0f","Type":"ContainerDied","Data":"8f03f21f82733255a91cbd54285e6dd2052f5e712ecaf862805308a0909829c4"}
Oct 30 21:33:04.401490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.401487    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f03f21f82733255a91cbd54285e6dd2052f5e712ecaf862805308a0909829c4"
Oct 30 21:33:04.401833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.401815    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:33:04.448300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.447502    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume\") pod \"17679a52-0f73-429f-93bb-681ef7a13b0f\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") "
Oct 30 21:33:04.448300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.447574    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj\" (UniqueName: \"kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj\") pod \"17679a52-0f73-429f-93bb-681ef7a13b0f\" (UID: \"17679a52-0f73-429f-93bb-681ef7a13b0f\") "
Oct 30 21:33:04.467103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.467060    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume" pod "17679a52-0f73-429f-93bb-681ef7a13b0f" (UID: "17679a52-0f73-429f-93bb-681ef7a13b0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:04.468461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.468424    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj" pod "17679a52-0f73-429f-93bb-681ef7a13b0f" (UID: "17679a52-0f73-429f-93bb-681ef7a13b0f"). InnerVolumeSpecName "kube-api-access-tdldj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:04.566140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.550229    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17679a52-0f73-429f-93bb-681ef7a13b0f-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:04.566140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.561780    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdldj\" (UniqueName: \"kubernetes.io/projected/17679a52-0f73-429f-93bb-681ef7a13b0f-kube-api-access-tdldj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:04.752687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.742762    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 21:33:04.752687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.742842    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 21:33:04.760909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.760880    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:33:04.761338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:04.761318    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:33:05.411922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.411889    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"
Oct 30 21:33:05.461923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.461876    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"]
Oct 30 21:33:05.471258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.471224    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-6b588cdd4c-s67sb"]
Oct 30 21:33:05.716284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.716251    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:33:05.716716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.716700    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:33:05.717470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.717450    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:33:05.760533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.760485    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:33:05.760926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.760901    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:33:05.761906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:05.761889    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:33:07.214793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:07.214759    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17679a52-0f73-429f-93bb-681ef7a13b0f" path="/var/lib/kubelet/pods/17679a52-0f73-429f-93bb-681ef7a13b0f/volumes"
Oct 30 21:33:09.242323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:09.242283    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"]
Oct 30 21:33:10.288295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.288256    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"]
Oct 30 21:33:10.302736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.302694    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy"
Oct 30 21:33:10.302981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.302958    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy"
Oct 30 21:33:10.303110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303096    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3f853e10-748b-4fca-acab-bd324a88b1f1" containerName="payload-processing"
Oct 30 21:33:10.303194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303182    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f853e10-748b-4fca-acab-bd324a88b1f1" containerName="payload-processing"
Oct 30 21:33:10.303282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303272    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ff3c3028-b40a-4e77-8826-59b7efe0f6e4" containerName="payload-pre-processing"
Oct 30 21:33:10.303363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303352    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3c3028-b40a-4e77-8826-59b7efe0f6e4" containerName="payload-pre-processing"
Oct 30 21:33:10.303461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303450    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy"
Oct 30 21:33:10.303544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303532    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy"
Oct 30 21:33:10.303678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303665    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy"
Oct 30 21:33:10.303770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303759    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy"
Oct 30 21:33:10.303894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303884    6695 cpu_manager.go:414] "containerMap: removing container" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy"
Oct 30 21:33:10.303974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.303963    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy"
Oct 30 21:33:10.304078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304068    6695 cpu_manager.go:414] "containerMap: removing container" podUID="17679a52-0f73-429f-93bb-681ef7a13b0f" containerName="payload-processing"
Oct 30 21:33:10.304160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304149    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17679a52-0f73-429f-93bb-681ef7a13b0f" containerName="payload-processing"
Oct 30 21:33:10.304509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304489    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="33268e64-f853-4363-9bbf-191ff48e9205" containerName="istio-proxy"
Oct 30 21:33:10.304665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304649    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="17679a52-0f73-429f-93bb-681ef7a13b0f" containerName="payload-processing"
Oct 30 21:33:10.304779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304766    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b64f18b-9634-486a-9ec2-a52afa35fd99" containerName="istio-proxy"
Oct 30 21:33:10.304879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304867    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f853e10-748b-4fca-acab-bd324a88b1f1" containerName="payload-processing"
Oct 30 21:33:10.304963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.304953    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3c3028-b40a-4e77-8826-59b7efe0f6e4" containerName="payload-pre-processing"
Oct 30 21:33:10.317723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.317690    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b69f28c-173a-497d-84c5-d07cce3a16fb" containerName="istio-proxy"
Oct 30 21:33:10.317970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.317953    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae9c26d-4245-4118-a693-8315be8781b9" containerName="istio-proxy"
Oct 30 21:33:10.343877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.343844    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"]
Oct 30 21:33:10.344201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.344186    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.446631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.363835    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3afc5e-dockercfg-4vn27\""
Oct 30 21:33:10.460691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.460649    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3afc5e\""
Oct 30 21:33:10.471497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.470489    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx\") pod \"payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.471497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.470670    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.591818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.591722    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.592013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.591901    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx\") pod \"payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.604147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.604113    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.624645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.620396    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"]
Oct 30 21:33:10.655566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.655536    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"]
Oct 30 21:33:10.655848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.655719    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.656035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.656013    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx\") pod \"payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.693369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.693256    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjxt\" (UniqueName: \"kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.694469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.694320    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.769204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.769159    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:10.795534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.795499    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjxt\" (UniqueName: \"kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.795747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.795581    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.796503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.796479    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.818090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.818056    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjxt\" (UniqueName: \"kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt\") pod \"payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:10.979416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:10.979380    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:11.145678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.142156    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"]
Oct 30 21:33:11.306867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.306840    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"]
Oct 30 21:33:11.309043 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:11.309010    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d5fce9_d075_4ece_849a_8212323a51a2.slice/crio-9b12cbce89959876057e1ff64050541587124995eca5e49ddf0ea0b934c79f6a WatchSource:0}: Error finding container 9b12cbce89959876057e1ff64050541587124995eca5e49ddf0ea0b934c79f6a: Status 404 returned error can't find the container with id 9b12cbce89959876057e1ff64050541587124995eca5e49ddf0ea0b934c79f6a
Oct 30 21:33:11.614646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.611012    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" event={"ID":"5374b820-82f7-47ab-a547-ea7a2563331c","Type":"ContainerStarted","Data":"35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003"}
Oct 30 21:33:11.614646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.611078    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" event={"ID":"5374b820-82f7-47ab-a547-ea7a2563331c","Type":"ContainerStarted","Data":"8e90e311db8fc002141a285c7c57388a257fce889a09232ad561bd7cac9ffb24"}
Oct 30 21:33:11.614646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.612145    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:11.638628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.637823    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" event={"ID":"78d5fce9-d075-4ece-849a-8212323a51a2","Type":"ContainerStarted","Data":"474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2"}
Oct 30 21:33:11.638628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.637887    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" event={"ID":"78d5fce9-d075-4ece-849a-8212323a51a2","Type":"ContainerStarted","Data":"9b12cbce89959876057e1ff64050541587124995eca5e49ddf0ea0b934c79f6a"}
Oct 30 21:33:11.638846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.638696    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:11.674396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:11.674286    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" podStartSLOduration=1.6742727130000001 podStartE2EDuration="1.674272713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:11.671718648 +0000 UTC m=+1374.594176947" watchObservedRunningTime="2026-10-30 21:33:11.674272713 +0000 UTC m=+1374.596731001"
Oct 30 21:33:14.023131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.023071    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" podStartSLOduration=4.023055408 podStartE2EDuration="4.023055408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:11.738852981 +0000 UTC m=+1374.661311280" watchObservedRunningTime="2026-10-30 21:33:14.023055408 +0000 UTC m=+1376.945513754"
Oct 30 21:33:14.026199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.026166    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"]
Oct 30 21:33:14.036536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.036505    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5038dc8-4tvwj"]
Oct 30 21:33:14.242837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.242730    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj" podUID="bfa73104-7772-43e0-a79d-85d7de79d72c" containerName="payload-pre-processing" containerID="cri-o://117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595" gracePeriod=25
Oct 30 21:33:14.626593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.626250    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:33:14.665043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.664985    6695 generic.go:356] "Generic (PLEG): container finished" podID="bfa73104-7772-43e0-a79d-85d7de79d72c" containerID="117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595" exitCode=0
Oct 30 21:33:14.665043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.665038    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"
Oct 30 21:33:14.665331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.665081    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj" event={"ID":"bfa73104-7772-43e0-a79d-85d7de79d72c","Type":"ContainerDied","Data":"117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595"}
Oct 30 21:33:14.665331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.665140    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj" event={"ID":"bfa73104-7772-43e0-a79d-85d7de79d72c","Type":"ContainerDied","Data":"af9f2128d80078df65f07f6464f52aa7f9d13176ae59b6add634513c3ad7485a"}
Oct 30 21:33:14.665331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.665169    6695 scope.go:122] "RemoveContainer" containerID="117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595"
Oct 30 21:33:14.677915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.677892    6695 scope.go:122] "RemoveContainer" containerID="117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595"
Oct 30 21:33:14.678343 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:33:14.678271    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595\": container with ID starting with 117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595 not found: ID does not exist" containerID="117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595"
Oct 30 21:33:14.678343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.678313    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595"} err="failed to get container status \"117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595\": rpc error: code = NotFound desc = could not find container \"117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595\": container with ID starting with 117e2ac468c6cbcb4318ccfa543fa1692712298693421b163f4b6ab7ac9a0595 not found: ID does not exist"
Oct 30 21:33:14.701108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.701076    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume\") pod \"bfa73104-7772-43e0-a79d-85d7de79d72c\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") "
Oct 30 21:33:14.701272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.701144    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff\" (UniqueName: \"kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff\") pod \"bfa73104-7772-43e0-a79d-85d7de79d72c\" (UID: \"bfa73104-7772-43e0-a79d-85d7de79d72c\") "
Oct 30 21:33:14.701537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.701508    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume" pod "bfa73104-7772-43e0-a79d-85d7de79d72c" (UID: "bfa73104-7772-43e0-a79d-85d7de79d72c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:14.701739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.701723    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa73104-7772-43e0-a79d-85d7de79d72c-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:14.703276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.703243    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff" pod "bfa73104-7772-43e0-a79d-85d7de79d72c" (UID: "bfa73104-7772-43e0-a79d-85d7de79d72c"). InnerVolumeSpecName "kube-api-access-sjqff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:14.803089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:14.803045    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqff\" (UniqueName: \"kubernetes.io/projected/bfa73104-7772-43e0-a79d-85d7de79d72c-kube-api-access-sjqff\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:15.011394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:15.011359    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"]
Oct 30 21:33:15.015724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:15.015687    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-6655dbc5f5-l7kpj"]
Oct 30 21:33:15.215899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:15.215822    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf4386a-b018-455f-b823-d83c049d1ea5" path="/var/lib/kubelet/pods/8bf4386a-b018-455f-b823-d83c049d1ea5/volumes"
Oct 30 21:33:15.216438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:15.216411    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa73104-7772-43e0-a79d-85d7de79d72c" path="/var/lib/kubelet/pods/bfa73104-7772-43e0-a79d-85d7de79d72c/volumes"
Oct 30 21:33:17.737810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:17.736324    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": read tcp 10.134.0.2:41162->10.134.0.156:15021: read: connection reset by peer" start-of-body=
Oct 30 21:33:17.737810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:17.736400    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": read tcp 10.134.0.2:41162->10.134.0.156:15021: read: connection reset by peer"
Oct 30 21:33:18.472926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.472888    6695 scope.go:122] "RemoveContainer" containerID="8c97863cea9534daa350d80971aacdad86813508dbb78add6c189392e384cbbe"
Oct 30 21:33:18.487165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.487106    6695 scope.go:122] "RemoveContainer" containerID="d38b92c52e2e695e49face1fa3f217f328cb654099fc03a5064f7634ff3f166d"
Oct 30 21:33:18.498268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.498248    6695 scope.go:122] "RemoveContainer" containerID="571447f1ce42db87d640a9f8d9bd22cb0c26d6ceb7f6c395f369467f9e453c12"
Oct 30 21:33:18.508502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.508464    6695 scope.go:122] "RemoveContainer" containerID="29432fa378b18d20eb63fa97011e067726d96af633d3e63e7d1dace4965ec850"
Oct 30 21:33:18.517553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.517531    6695 scope.go:122] "RemoveContainer" containerID="70c632da9c99767c5f843eccbf0e1ffb8b898c423943d0ea1a08eff8700fef2f"
Oct 30 21:33:18.526634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.526596    6695 scope.go:122] "RemoveContainer" containerID="22c1ca597bf2cdbd2f88cbb781ebc98701598e2137a1c79ac0b5df9c1b026f09"
Oct 30 21:33:18.535950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.535933    6695 scope.go:122] "RemoveContainer" containerID="b0e06407607516503b71be5c3a4af30198bd85a24325e019e16096b9914592d0"
Oct 30 21:33:18.720501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.720468    6695 generic.go:356] "Generic (PLEG): container finished" podID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerID="0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793" exitCode=0
Oct 30 21:33:18.720704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.720537    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" event={"ID":"8e93ab7d-f44a-4035-b945-63ba14a87d0d","Type":"ContainerDied","Data":"0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"}
Oct 30 21:33:18.721045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.721027    6695 scope.go:122] "RemoveContainer" containerID="0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"
Oct 30 21:33:18.722674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.722647    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:18.722758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.722714    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:18.722758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:18.722745    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:19.743047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:19.743011    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" event={"ID":"8e93ab7d-f44a-4035-b945-63ba14a87d0d","Type":"ContainerStarted","Data":"981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324"}
Oct 30 21:33:20.065637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.063691    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"]
Oct 30 21:33:20.065637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.064618    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bfa73104-7772-43e0-a79d-85d7de79d72c" containerName="payload-pre-processing"
Oct 30 21:33:20.065637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.064639    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa73104-7772-43e0-a79d-85d7de79d72c" containerName="payload-pre-processing"
Oct 30 21:33:20.065637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.064774    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa73104-7772-43e0-a79d-85d7de79d72c" containerName="payload-pre-processing"
Oct 30 21:33:20.068108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.068091    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.083914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.083878    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a1ee3-dockercfg-rc4gw\""
Oct 30 21:33:20.083914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.083895    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a1ee3\""
Oct 30 21:33:20.087240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.087202    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"]
Oct 30 21:33:20.294631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.289004    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jgg\" (UniqueName: \"kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.294631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.289072    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.399689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.390491    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jgg\" (UniqueName: \"kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.399689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.398386    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.401274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.401241    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.402910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.402884    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"]
Oct 30 21:33:20.408861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.408830    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"]
Oct 30 21:33:20.409005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.408953    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.444452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.444160    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jgg\" (UniqueName: \"kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.500249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.500212    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.500476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.500284    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmgsb\" (UniqueName: \"kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb\") pod \"payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.588282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.588247    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:33:20.588474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.588450    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:33:20.588529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.588503    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:33:20.601700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.601667    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmgsb\" (UniqueName: \"kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb\") pod \"payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.601977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.601956    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.602708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.602691    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.629628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.622633    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmgsb\" (UniqueName: \"kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb\") pod \"payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:20.684261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.684178    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:20.751910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:20.751542    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:21.037696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.034539    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"]
Oct 30 21:33:21.071036 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:21.069687    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b11200d_d525_4395_8fbc_220b70efaeea.slice/crio-d8ac706e9cd9ef195efd0704e5fb8c88130f5b3ccf88aca90c13b44076ecaef4 WatchSource:0}: Error finding container d8ac706e9cd9ef195efd0704e5fb8c88130f5b3ccf88aca90c13b44076ecaef4: Status 404 returned error can't find the container with id d8ac706e9cd9ef195efd0704e5fb8c88130f5b3ccf88aca90c13b44076ecaef4
Oct 30 21:33:21.152777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.152728    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"]
Oct 30 21:33:21.157340 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:21.157314    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bb5c7f_d9f7_4c9b_a830_93d806eaaf4d.slice/crio-c69d04cf5a88a4e6426ad983deec4dc7d68bd831105f9f99628cba673bfcda11 WatchSource:0}: Error finding container c69d04cf5a88a4e6426ad983deec4dc7d68bd831105f9f99628cba673bfcda11: Status 404 returned error can't find the container with id c69d04cf5a88a4e6426ad983deec4dc7d68bd831105f9f99628cba673bfcda11
Oct 30 21:33:21.588303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.588270    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:33:21.588502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.588319    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:33:21.762126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.761814    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh" event={"ID":"8b11200d-d525-4395-8fbc-220b70efaeea","Type":"ContainerStarted","Data":"3198703d64c987a1005ae48a3cfd031d0f1b92bd1454b27caee8b4f0493c5117"}
Oct 30 21:33:21.762622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.762161    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh" event={"ID":"8b11200d-d525-4395-8fbc-220b70efaeea","Type":"ContainerStarted","Data":"d8ac706e9cd9ef195efd0704e5fb8c88130f5b3ccf88aca90c13b44076ecaef4"}
Oct 30 21:33:21.762622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.762193    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:21.763872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.763757    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd" event={"ID":"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d","Type":"ContainerStarted","Data":"204e10341fe678156365ff235360afd28b0f952ff13bce28e82ca2fc3f0dce08"}
Oct 30 21:33:21.763872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.763811    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd" event={"ID":"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d","Type":"ContainerStarted","Data":"c69d04cf5a88a4e6426ad983deec4dc7d68bd831105f9f99628cba673bfcda11"}
Oct 30 21:33:21.763872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.763842    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:21.798802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:21.798749    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh" podStartSLOduration=1.7987351679999999 podStartE2EDuration="1.798735168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:21.797146376 +0000 UTC m=+1384.719604673" watchObservedRunningTime="2026-10-30 21:33:21.798735168 +0000 UTC m=+1384.721193491"
Oct 30 21:33:22.588007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:22.587974    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:33:22.588192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:22.588027    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:33:23.589111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:23.589070    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:33:23.589759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:23.589728    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:33:23.673910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:23.672826    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:33:23.706353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:23.706318    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:33:23.863251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:23.863115    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd" podStartSLOduration=3.8630975100000002 podStartE2EDuration="3.86309751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:21.839295642 +0000 UTC m=+1384.761753966" watchObservedRunningTime="2026-10-30 21:33:23.86309751 +0000 UTC m=+1386.785555808"
Oct 30 21:33:24.620050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:24.619647    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:33:24.620050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:24.619703    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:33:25.588382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:25.588343    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:33:25.588579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:25.588403    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:33:26.591907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:26.591874    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:33:26.592378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:26.592175    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:33:26.592883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:26.592866    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:33:27.421060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.420913    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"]
Oct 30 21:33:27.430480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.430453    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.447993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.447968    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-38e5bf-openshift-default-dockercfg-q2kkj\""
Oct 30 21:33:27.460531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.460495    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"]
Oct 30 21:33:27.574211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574173    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574233    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574259    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574295    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574339    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574359    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574376    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574440    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.574781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.574476    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxkd\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.691677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.684151    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.691677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.684271    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxkd\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.691677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.689850    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.691677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.689978    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.692860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.692791    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.700642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.695570    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.700642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.697580    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.700642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.697687    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.700642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.697789    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.700642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.697819    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.700642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.697850    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.701636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.701587    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.705266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.701852    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.705266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.705176    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.719628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.717327    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.734626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.730020    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.744636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.737147    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.754083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.754047    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxkd\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd\") pod \"e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") " pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:27.777933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:27.777897    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:28.140973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.140939    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"]
Oct 30 21:33:28.142805 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:28.142774    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7dd11d_17cb_4ad5_b7ff_4341cfd9d0ba.slice/crio-a85bec02ef3fd7b8b9fa22427e8237cf8c1fc51fcd77901a76d4d0104c92d0d3 WatchSource:0}: Error finding container a85bec02ef3fd7b8b9fa22427e8237cf8c1fc51fcd77901a76d4d0104c92d0d3: Status 404 returned error can't find the container with id a85bec02ef3fd7b8b9fa22427e8237cf8c1fc51fcd77901a76d4d0104c92d0d3
Oct 30 21:33:28.145631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.145569    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:28.145717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.145676    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:28.145717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.145709    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:28.168812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.168789    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"]
Oct 30 21:33:28.174661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.174640    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.189435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.189406    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:33:28.189550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.189403    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:33:28.189748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.189731    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:33:28.191642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.191616    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"]
Oct 30 21:33:28.198153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.198131    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:33:28.306501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.306458    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.306663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.306509    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.306811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.306771    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4mz\" (UniqueName: \"kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.408756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.408670    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.408756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.408720    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.409018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.408834    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4mz\" (UniqueName: \"kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.409428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.409407    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.419636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.417029    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.437356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.437322    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4mz\" (UniqueName: \"kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz\") pod \"maas-api-revoke-keys-e2e-api-a-18a043-q2h9d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.538025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.537981    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:28.874716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:28.874684    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"]
Oct 30 21:33:28.878302 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:28.878271    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e071fb1_c954_42fb_9237_ca9923b43d6d.slice/crio-cd5ae8b1eb01859a7731ffd5d2b6fafc4092159dc4be82ee87b63b44dbf33865 WatchSource:0}: Error finding container cd5ae8b1eb01859a7731ffd5d2b6fafc4092159dc4be82ee87b63b44dbf33865: Status 404 returned error can't find the container with id cd5ae8b1eb01859a7731ffd5d2b6fafc4092159dc4be82ee87b63b44dbf33865
Oct 30 21:33:29.009980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:29.009932    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" event={"ID":"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba","Type":"ContainerStarted","Data":"73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"}
Oct 30 21:33:29.010188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:29.010004    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" event={"ID":"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba","Type":"ContainerStarted","Data":"a85bec02ef3fd7b8b9fa22427e8237cf8c1fc51fcd77901a76d4d0104c92d0d3"}
Oct 30 21:33:29.042182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:29.042141    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d" event={"ID":"7e071fb1-c954-42fb-9237-ca9923b43d6d","Type":"ContainerStarted","Data":"cd5ae8b1eb01859a7731ffd5d2b6fafc4092159dc4be82ee87b63b44dbf33865"}
Oct 30 21:33:29.055730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:29.055675    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podStartSLOduration=2.055660581 podStartE2EDuration="2.055660581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:29.052118372 +0000 UTC m=+1391.974576680" watchObservedRunningTime="2026-10-30 21:33:29.055660581 +0000 UTC m=+1391.978118878"
Oct 30 21:33:29.779157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:29.779117    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:29.784859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:29.784828    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:30.052110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:30.052077    6695 generic.go:356] "Generic (PLEG): container finished" podID="7e071fb1-c954-42fb-9237-ca9923b43d6d" containerID="09169b24b4d5add2f65dfc7f9162b6cfadb089282b1b02dc15895cae590d404c" exitCode=0
Oct 30 21:33:30.052495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:30.052158    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d" event={"ID":"7e071fb1-c954-42fb-9237-ca9923b43d6d","Type":"ContainerDied","Data":"09169b24b4d5add2f65dfc7f9162b6cfadb089282b1b02dc15895cae590d404c"}
Oct 30 21:33:30.052615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:30.052585    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:30.053683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:30.053663    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:33:31.077262 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:33:31.077217    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:33:31.219672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.219642    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:31.336938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.336849    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz\" (UniqueName: \"kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz\") pod \"7e071fb1-c954-42fb-9237-ca9923b43d6d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") "
Oct 30 21:33:31.337101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.336982    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca\") pod \"7e071fb1-c954-42fb-9237-ca9923b43d6d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") "
Oct 30 21:33:31.337101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.337071    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca\") pod \"7e071fb1-c954-42fb-9237-ca9923b43d6d\" (UID: \"7e071fb1-c954-42fb-9237-ca9923b43d6d\") "
Oct 30 21:33:31.337489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.337454    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca" pod "7e071fb1-c954-42fb-9237-ca9923b43d6d" (UID: "7e071fb1-c954-42fb-9237-ca9923b43d6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:31.337898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.337875    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:31.339156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.339122    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz" pod "7e071fb1-c954-42fb-9237-ca9923b43d6d" (UID: "7e071fb1-c954-42fb-9237-ca9923b43d6d"). InnerVolumeSpecName "kube-api-access-6z4mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:31.339293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.339269    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca" pod "7e071fb1-c954-42fb-9237-ca9923b43d6d" (UID: "7e071fb1-c954-42fb-9237-ca9923b43d6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:31.440003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.439959    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z4mz\" (UniqueName: \"kubernetes.io/projected/7e071fb1-c954-42fb-9237-ca9923b43d6d-kube-api-access-6z4mz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:31.440003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:31.440007    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e071fb1-c954-42fb-9237-ca9923b43d6d-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:32.071344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:32.071310    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"
Oct 30 21:33:32.071519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:32.071341    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d" event={"ID":"7e071fb1-c954-42fb-9237-ca9923b43d6d","Type":"ContainerDied","Data":"cd5ae8b1eb01859a7731ffd5d2b6fafc4092159dc4be82ee87b63b44dbf33865"}
Oct 30 21:33:32.071519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:32.071429    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5ae8b1eb01859a7731ffd5d2b6fafc4092159dc4be82ee87b63b44dbf33865"
Oct 30 21:33:32.774158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:32.774127    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:33:32.774704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:32.774201    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:33:33.249030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.248945    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"]
Oct 30 21:33:33.249624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.249595    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7e071fb1-c954-42fb-9237-ca9923b43d6d" containerName="revoke-keys"
Oct 30 21:33:33.249686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.249625    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e071fb1-c954-42fb-9237-ca9923b43d6d" containerName="revoke-keys"
Oct 30 21:33:33.249766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.249756    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e071fb1-c954-42fb-9237-ca9923b43d6d" containerName="revoke-keys"
Oct 30 21:33:33.252790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.252763    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.266739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.266702    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38e5bf\""
Oct 30 21:33:33.272065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.272039    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38e5bf-dockercfg-fpnpm\""
Oct 30 21:33:33.272277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.272183    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"]
Oct 30 21:33:33.428151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.425915    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56g4n\" (UniqueName: \"kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.428151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.426027    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.445628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.443805    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"]
Oct 30 21:33:33.506678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.504400    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"]
Oct 30 21:33:33.506678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.504539    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.547628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.535415    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56g4n\" (UniqueName: \"kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.549093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.549061    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.550511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.550457    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.579867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.579591    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56g4n\" (UniqueName: \"kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:33.650263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.650205    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.650472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.650333    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lncc5\" (UniqueName: \"kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5\") pod \"payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.751370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.751336    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.751560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.751399    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncc5\" (UniqueName: \"kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5\") pod \"payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.752149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.752127    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.770117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.770084    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lncc5\" (UniqueName: \"kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5\") pod \"payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.843670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.843636    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:33.874494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:33.874455    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:34.155903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:34.155856    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"]
Oct 30 21:33:34.158534 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:34.158508    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aee02cf_0177_488b_a83a_8571222237ea.slice/crio-950d640adcc03ef960eae64b3a1be5d26ca6e8b525528a4b5c3244c6e059a054 WatchSource:0}: Error finding container 950d640adcc03ef960eae64b3a1be5d26ca6e8b525528a4b5c3244c6e059a054: Status 404 returned error can't find the container with id 950d640adcc03ef960eae64b3a1be5d26ca6e8b525528a4b5c3244c6e059a054
Oct 30 21:33:34.197711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:34.197677    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"]
Oct 30 21:33:34.201319 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:34.201292    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372e3a8e_7406_4c82_a782_2da39fb68c74.slice/crio-e3bf9b22fa9f326a060a4215b1863aa687d856c3b73c0e859940573a42cbff78 WatchSource:0}: Error finding container e3bf9b22fa9f326a060a4215b1863aa687d856c3b73c0e859940573a42cbff78: Status 404 returned error can't find the container with id e3bf9b22fa9f326a060a4215b1863aa687d856c3b73c0e859940573a42cbff78
Oct 30 21:33:35.094394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.094346    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb" event={"ID":"372e3a8e-7406-4c82-a782-2da39fb68c74","Type":"ContainerStarted","Data":"65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d"}
Oct 30 21:33:35.094989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.094410    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb" event={"ID":"372e3a8e-7406-4c82-a782-2da39fb68c74","Type":"ContainerStarted","Data":"e3bf9b22fa9f326a060a4215b1863aa687d856c3b73c0e859940573a42cbff78"}
Oct 30 21:33:35.094989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.094442    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:35.096335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.096304    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92" event={"ID":"0aee02cf-0177-488b-a83a-8571222237ea","Type":"ContainerStarted","Data":"3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d"}
Oct 30 21:33:35.096475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.096360    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:35.096475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.096384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92" event={"ID":"0aee02cf-0177-488b-a83a-8571222237ea","Type":"ContainerStarted","Data":"950d640adcc03ef960eae64b3a1be5d26ca6e8b525528a4b5c3244c6e059a054"}
Oct 30 21:33:35.097942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.097920    6695 generic.go:356] "Generic (PLEG): container finished" podID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerID="953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442" exitCode=0
Oct 30 21:33:35.098046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.097986    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" event={"ID":"22ba006c-a1ed-4618-81b6-91144a0c50a3","Type":"ContainerDied","Data":"953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"}
Oct 30 21:33:35.098333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.098318    6695 scope.go:122] "RemoveContainer" containerID="953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"
Oct 30 21:33:35.099783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.099757    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:35.099864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.099822    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:35.099864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.099853    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:35.130584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.130535    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb" podStartSLOduration=2.130522681 podStartE2EDuration="2.130522681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:35.129863655 +0000 UTC m=+1398.052321954" watchObservedRunningTime="2026-10-30 21:33:35.130522681 +0000 UTC m=+1398.052980979"
Oct 30 21:33:35.208653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:35.205639    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92" podStartSLOduration=2.205617497 podStartE2EDuration="2.205617497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:35.203004153 +0000 UTC m=+1398.125462458" watchObservedRunningTime="2026-10-30 21:33:35.205617497 +0000 UTC m=+1398.128075796"
Oct 30 21:33:36.184623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.160894    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"]
Oct 30 21:33:36.184623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.181832    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"]
Oct 30 21:33:36.184623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.181889    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" event={"ID":"22ba006c-a1ed-4618-81b6-91144a0c50a3","Type":"ContainerStarted","Data":"fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0"}
Oct 30 21:33:36.184623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.183249    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.263012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.262971    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.263726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.263679    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vjw\" (UniqueName: \"kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.292145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.292105    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7222e2\""
Oct 30 21:33:36.292991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.292895    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7222e2-dockercfg-82pqw\""
Oct 30 21:33:36.367103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.365670    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vjw\" (UniqueName: \"kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.367103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.366115    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.367103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.366948    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.384513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.384476    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vjw\" (UniqueName: \"kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw\") pod \"payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.607722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.607692    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:36.772421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.772387    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"]
Oct 30 21:33:36.777188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.776670    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.791380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.791333    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b0e991-openshift-default-dockercfg-nnxfc\""
Oct 30 21:33:36.807321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.807232    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"]
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881635    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881718    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881749    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwhk\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881797    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881881    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881933    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.881966    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.882006    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.882106    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:36.918836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.882729    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:33:36.931628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.921352    6695 patch_prober.go:29] interesting pod/e2e-api-b-b1659a-openshift-default-594764768c-v58qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 21:33:36.931628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:36.921413    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 21:33:37.097848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.014651    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.098356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.098313    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.098627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.016290    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.098627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.098555    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwhk\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.099001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.098925    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.099470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.099358    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.099940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.099681    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.099940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.099890    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.100313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.100277    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.100849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.100811    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.101040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.100971    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.102450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.102184    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.103084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.103027    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.103519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.103465    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.190622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.190527    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"]
Oct 30 21:33:37.205648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.200901    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.205648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.201118    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.205648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.201529    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.279138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.279100    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwhk\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk\") pod \"e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") " pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.326637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.321812    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r" event={"ID":"69bdea97-70db-461b-9e7b-16e382927942","Type":"ContainerStarted","Data":"e3bcd8f33c16dabfb1b8a06e96f92988693a06a8553345f4d7e492cec29d5c98"}
Oct 30 21:33:37.403564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.403347    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:37.847059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.847021    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"]
Oct 30 21:33:37.847847 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:37.847818    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1730f8_e0e3_4345_9952_cde0a41a63b5.slice/crio-c68b394f5b56b022eb8ac8fbe9cfdbe37df826f981d46fad05b1f8cbb218d13e WatchSource:0}: Error finding container c68b394f5b56b022eb8ac8fbe9cfdbe37df826f981d46fad05b1f8cbb218d13e: Status 404 returned error can't find the container with id c68b394f5b56b022eb8ac8fbe9cfdbe37df826f981d46fad05b1f8cbb218d13e
Oct 30 21:33:37.851219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.851171    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:37.851317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.851267    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:37.851364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.851313    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:33:37.881132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.881098    6695 patch_prober.go:29] interesting pod/e2e-api-b-b1659a-openshift-default-594764768c-v58qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 21:33:37.881257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:37.881162    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 21:33:38.306458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.306417    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r" event={"ID":"69bdea97-70db-461b-9e7b-16e382927942","Type":"ContainerStarted","Data":"4ca0522d4fab920be357e744ef85a1c89f98d3d0f20f22e2b7ae13c10d1867f3"}
Oct 30 21:33:38.306986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.306513    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:38.308594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.308564    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" event={"ID":"7c1730f8-e0e3-4345-9952-cde0a41a63b5","Type":"ContainerStarted","Data":"8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"}
Oct 30 21:33:38.308764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.308647    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" event={"ID":"7c1730f8-e0e3-4345-9952-cde0a41a63b5","Type":"ContainerStarted","Data":"c68b394f5b56b022eb8ac8fbe9cfdbe37df826f981d46fad05b1f8cbb218d13e"}
Oct 30 21:33:38.344739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.344669    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r" podStartSLOduration=2.344650559 podStartE2EDuration="2.344650559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:38.343590131 +0000 UTC m=+1401.266048428" watchObservedRunningTime="2026-10-30 21:33:38.344650559 +0000 UTC m=+1401.267108854"
Oct 30 21:33:38.382781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.382724    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podStartSLOduration=2.382708463 podStartE2EDuration="2.382708463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:38.382296121 +0000 UTC m=+1401.304754420" watchObservedRunningTime="2026-10-30 21:33:38.382708463 +0000 UTC m=+1401.305166760"
Oct 30 21:33:38.403678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.403647    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:38.409843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.409816    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:38.898629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.896631    6695 patch_prober.go:29] interesting pod/e2e-api-b-b1659a-openshift-default-594764768c-v58qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 21:33:38.898629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:38.896694    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 21:33:39.347635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:39.347515    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:39.356636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:39.353856    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:33:39.881736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:39.881692    6695 patch_prober.go:29] interesting pod/e2e-api-b-b1659a-openshift-default-594764768c-v58qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 21:33:39.881934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:39.881747    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 21:33:40.882037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:40.881995    6695 patch_prober.go:29] interesting pod/e2e-api-b-b1659a-openshift-default-594764768c-v58qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 21:33:40.882513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:40.882056    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 21:33:41.882137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:41.882098    6695 patch_prober.go:29] interesting pod/e2e-api-b-b1659a-openshift-default-594764768c-v58qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 30 21:33:41.882673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:41.882148    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 30 21:33:42.885888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:42.885852    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:33:42.886403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:42.886377    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:33:42.887298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:42.887278    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:33:43.182963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.182863    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"]
Oct 30 21:33:43.187657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.187535    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"]
Oct 30 21:33:43.187830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.187680    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.205574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.205547    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b0e991-dockercfg-845x7\""
Oct 30 21:33:43.205797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.205776    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0e991\""
Oct 30 21:33:43.293134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.293099    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jscnp\" (UniqueName: \"kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp\") pod \"payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.293374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.293351    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.394425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.394389    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jscnp\" (UniqueName: \"kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp\") pod \"payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.394620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.394568    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.395294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.395271    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.413124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.413086    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jscnp\" (UniqueName: \"kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp\") pod \"payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.509143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.509059    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:43.811597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:43.811566    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"]
Oct 30 21:33:43.816456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:33:43.816422    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14e2126_7387_4d5d_9aa4_69f31f8c0c57.slice/crio-62cef96d7c2cdf56e8a6a91083bb63f1656cde2d3e5d7a367f5d321d70563b3e WatchSource:0}: Error finding container 62cef96d7c2cdf56e8a6a91083bb63f1656cde2d3e5d7a367f5d321d70563b3e: Status 404 returned error can't find the container with id 62cef96d7c2cdf56e8a6a91083bb63f1656cde2d3e5d7a367f5d321d70563b3e
Oct 30 21:33:44.444996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:44.444913    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j" event={"ID":"e14e2126-7387-4d5d-9aa4-69f31f8c0c57","Type":"ContainerStarted","Data":"94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e"}
Oct 30 21:33:44.445765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:44.445738    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:44.445896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:44.445778    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j" event={"ID":"e14e2126-7387-4d5d-9aa4-69f31f8c0c57","Type":"ContainerStarted","Data":"62cef96d7c2cdf56e8a6a91083bb63f1656cde2d3e5d7a367f5d321d70563b3e"}
Oct 30 21:33:44.485547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:44.485487    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j" podStartSLOduration=2.485472695 podStartE2EDuration="2.485472695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:44.482277689 +0000 UTC m=+1407.404735999" watchObservedRunningTime="2026-10-30 21:33:44.485472695 +0000 UTC m=+1407.407930993"
Oct 30 21:33:46.163454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:46.163418    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:33:46.163979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:46.163662    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:33:46.731074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:46.731035    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"]
Oct 30 21:33:46.731307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:46.731286    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy" containerID="cri-o://09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864" gracePeriod=30
Oct 30 21:33:49.355722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:49.354417    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:33:52.003788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.003722    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:33:52.061365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061326    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.061566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061382    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.061566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061465    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.061566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061511    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.061757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061680    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.061757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061704    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:52.061869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061752    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:52.061869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061780    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.061991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.061975    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.062050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.062008    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.062105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.062070    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:52.062171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.062158    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket\") pod \"1be073e4-66f5-4635-b73c-d9901f1c155e\" (UID: \"1be073e4-66f5-4635-b73c-d9901f1c155e\") "
Oct 30 21:33:52.062655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.062395    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:52.062655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.062580    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:52.063198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.063170    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be073e4-66f5-4635-b73c-d9901f1c155e-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.063198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.063198    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.063385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.063214    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.063385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.063226    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.063385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.063239    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.064348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.064308    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:33:52.064769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.064738    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "kube-api-access-v82jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:52.064999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.064970    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:52.065860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.065837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token" pod "1be073e4-66f5-4635-b73c-d9901f1c155e" (UID: "1be073e4-66f5-4635-b73c-d9901f1c155e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:52.164470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.164434    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.164470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.164468    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.164470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.164480    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be073e4-66f5-4635-b73c-d9901f1c155e-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.164763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.164493    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82jv\" (UniqueName: \"kubernetes.io/projected/1be073e4-66f5-4635-b73c-d9901f1c155e-kube-api-access-v82jv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:52.585912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.585874    6695 generic.go:356] "Generic (PLEG): container finished" podID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerID="09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864" exitCode=0
Oct 30 21:33:52.586116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.585916    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" event={"ID":"1be073e4-66f5-4635-b73c-d9901f1c155e","Type":"ContainerDied","Data":"09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864"}
Oct 30 21:33:52.586116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.585964    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl" event={"ID":"1be073e4-66f5-4635-b73c-d9901f1c155e","Type":"ContainerDied","Data":"9d6e0986cb8946c4ec617afb4222aa38f14a020087266d3912412f0296233474"}
Oct 30 21:33:52.586116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.585922    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"
Oct 30 21:33:52.586116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.585991    6695 scope.go:122] "RemoveContainer" containerID="09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864"
Oct 30 21:33:52.597246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.597224    6695 scope.go:122] "RemoveContainer" containerID="09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864"
Oct 30 21:33:52.597546 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:33:52.597521    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864\": container with ID starting with 09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864 not found: ID does not exist" containerID="09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864"
Oct 30 21:33:52.597694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.597553    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864"} err="failed to get container status \"09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864\": rpc error: code = NotFound desc = could not find container \"09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864\": container with ID starting with 09fe5c2fa029a1be30efadfef94a53b4ca4fbcd1c50748a4dc12b87912971864 not found: ID does not exist"
Oct 30 21:33:52.649652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.649595    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"]
Oct 30 21:33:52.658990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.658957    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-j7szl"]
Oct 30 21:33:52.917792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.917715    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"]
Oct 30 21:33:52.918385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.918372    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy"
Oct 30 21:33:52.918429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.918387    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy"
Oct 30 21:33:52.918502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.918492    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" containerName="istio-proxy"
Oct 30 21:33:52.921160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.921143    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:52.936245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.936215    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:33:52.936245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.936216    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:33:52.936570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.936442    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:33:52.936694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.936672    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:33:52.940947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.940911    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"]
Oct 30 21:33:52.977095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.977057    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:52.977291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.977242    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227qw\" (UniqueName: \"kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:52.977370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:52.977346    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.079375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.079338    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.079780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.079443    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227qw\" (UniqueName: \"kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.079780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.079492    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.080082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.080057    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.082029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.082004    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.099827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.099776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227qw\" (UniqueName: \"kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw\") pod \"maas-api-revoke-keys-e2e-ipp-a-229795-l9h85\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.216818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.216737    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be073e4-66f5-4635-b73c-d9901f1c155e" path="/var/lib/kubelet/pods/1be073e4-66f5-4635-b73c-d9901f1c155e/volumes"
Oct 30 21:33:53.234857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.234820    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:53.545482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.545450    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"]
Oct 30 21:33:53.596579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:53.596548    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85" event={"ID":"7804c58d-700a-4604-920a-41f91d1f7663","Type":"ContainerStarted","Data":"e89cfa8144a4d4419856420a37a2682287a673e9e52ac6924b5dbe74c2613d70"}
Oct 30 21:33:54.606289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:54.606249    6695 generic.go:356] "Generic (PLEG): container finished" podID="7804c58d-700a-4604-920a-41f91d1f7663" containerID="08a826deb5a844dfaf4ae0d173018a8b6775b165d05e05cd46fa7f8c20338ba9" exitCode=0
Oct 30 21:33:54.606701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:54.606324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85" event={"ID":"7804c58d-700a-4604-920a-41f91d1f7663","Type":"ContainerDied","Data":"08a826deb5a844dfaf4ae0d173018a8b6775b165d05e05cd46fa7f8c20338ba9"}
Oct 30 21:33:55.024844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.024803    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"]
Oct 30 21:33:55.028016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.027988    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642e3c3e-ww445"]
Oct 30 21:33:55.232669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.232636    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2" path="/var/lib/kubelet/pods/3c6f073e-dec0-4abb-a4ae-eaf1dc7fafd2/volumes"
Oct 30 21:33:55.495099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.495010    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:33:55.776662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.776627    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:55.824953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.824921    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca\") pod \"7804c58d-700a-4604-920a-41f91d1f7663\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") "
Oct 30 21:33:55.825134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.824969    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca\") pod \"7804c58d-700a-4604-920a-41f91d1f7663\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") "
Oct 30 21:33:55.825134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.825095    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw\" (UniqueName: \"kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw\") pod \"7804c58d-700a-4604-920a-41f91d1f7663\" (UID: \"7804c58d-700a-4604-920a-41f91d1f7663\") "
Oct 30 21:33:55.825363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.825323    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca" pod "7804c58d-700a-4604-920a-41f91d1f7663" (UID: "7804c58d-700a-4604-920a-41f91d1f7663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:55.825738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.825716    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:55.827836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.827806    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw" pod "7804c58d-700a-4604-920a-41f91d1f7663" (UID: "7804c58d-700a-4604-920a-41f91d1f7663"). InnerVolumeSpecName "kube-api-access-227qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:55.827963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.827810    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca" pod "7804c58d-700a-4604-920a-41f91d1f7663" (UID: "7804c58d-700a-4604-920a-41f91d1f7663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:55.927162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.927130    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7804c58d-700a-4604-920a-41f91d1f7663-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:55.927162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:55.927159    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227qw\" (UniqueName: \"kubernetes.io/projected/7804c58d-700a-4604-920a-41f91d1f7663-kube-api-access-227qw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:33:56.624041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:56.624011    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"
Oct 30 21:33:56.624250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:56.624043    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85" event={"ID":"7804c58d-700a-4604-920a-41f91d1f7663","Type":"ContainerDied","Data":"e89cfa8144a4d4419856420a37a2682287a673e9e52ac6924b5dbe74c2613d70"}
Oct 30 21:33:56.624250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:33:56.624097    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89cfa8144a4d4419856420a37a2682287a673e9e52ac6924b5dbe74c2613d70"
Oct 30 21:34:01.078473 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:01.078433    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:34:01.645106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.645048    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"]
Oct 30 21:34:01.645840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.645824    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7804c58d-700a-4604-920a-41f91d1f7663" containerName="revoke-keys"
Oct 30 21:34:01.645915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.645842    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7804c58d-700a-4604-920a-41f91d1f7663" containerName="revoke-keys"
Oct 30 21:34:01.645976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.645967    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7804c58d-700a-4604-920a-41f91d1f7663" containerName="revoke-keys"
Oct 30 21:34:01.648511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.648488    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.668508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.668474    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"]
Oct 30 21:34:01.676341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.676311    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:34:01.676540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.676332    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:34:01.676540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.676342    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:34:01.676540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.676388    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:34:01.840672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.840629    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzx4\" (UniqueName: \"kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.840884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.840790    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.840884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.840846    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.942290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.942208    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzx4\" (UniqueName: \"kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.942447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.942306    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.942447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.942344    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.943002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.942980    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.944905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.944883    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.960633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.960588    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzx4\" (UniqueName: \"kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4\") pod \"maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:01.961734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:01.961708    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:02.266876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:02.266618    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"]
Oct 30 21:34:02.270594 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:34:02.270558    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c51fa3_af03_4bc2_9752_251300d7bfa4.slice/crio-bf6c00254fda4c3f5fcc83f5500c17398a7ba8e9df65589c7006779fc19ae829 WatchSource:0}: Error finding container bf6c00254fda4c3f5fcc83f5500c17398a7ba8e9df65589c7006779fc19ae829: Status 404 returned error can't find the container with id bf6c00254fda4c3f5fcc83f5500c17398a7ba8e9df65589c7006779fc19ae829
Oct 30 21:34:02.671330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:02.671294    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th" event={"ID":"a3c51fa3-af03-4bc2-9752-251300d7bfa4","Type":"ContainerStarted","Data":"f758e19284ef8b4d85735f7aadb2bbb395069e23de3b9d3c9a0849f44bac2a03"}
Oct 30 21:34:02.671500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:02.671368    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th" event={"ID":"a3c51fa3-af03-4bc2-9752-251300d7bfa4","Type":"ContainerStarted","Data":"bf6c00254fda4c3f5fcc83f5500c17398a7ba8e9df65589c7006779fc19ae829"}
Oct 30 21:34:03.680443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:03.680350    6695 generic.go:356] "Generic (PLEG): container finished" podID="a3c51fa3-af03-4bc2-9752-251300d7bfa4" containerID="f758e19284ef8b4d85735f7aadb2bbb395069e23de3b9d3c9a0849f44bac2a03" exitCode=0
Oct 30 21:34:03.680858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:03.680439    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th" event={"ID":"a3c51fa3-af03-4bc2-9752-251300d7bfa4","Type":"ContainerDied","Data":"f758e19284ef8b4d85735f7aadb2bbb395069e23de3b9d3c9a0849f44bac2a03"}
Oct 30 21:34:04.929050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.929020    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:04.986870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.986785    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca\") pod \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") "
Oct 30 21:34:04.987055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.986886    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca\") pod \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") "
Oct 30 21:34:04.987055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.986940    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4\" (UniqueName: \"kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4\") pod \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\" (UID: \"a3c51fa3-af03-4bc2-9752-251300d7bfa4\") "
Oct 30 21:34:04.987353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.987309    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca" pod "a3c51fa3-af03-4bc2-9752-251300d7bfa4" (UID: "a3c51fa3-af03-4bc2-9752-251300d7bfa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:04.987822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.987796    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:04.989107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.989071    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca" pod "a3c51fa3-af03-4bc2-9752-251300d7bfa4" (UID: "a3c51fa3-af03-4bc2-9752-251300d7bfa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:04.989667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:04.989640    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4" pod "a3c51fa3-af03-4bc2-9752-251300d7bfa4" (UID: "a3c51fa3-af03-4bc2-9752-251300d7bfa4"). InnerVolumeSpecName "kube-api-access-qnzx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:05.089673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:05.089596    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c51fa3-af03-4bc2-9752-251300d7bfa4-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:05.089673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:05.089665    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnzx4\" (UniqueName: \"kubernetes.io/projected/a3c51fa3-af03-4bc2-9752-251300d7bfa4-kube-api-access-qnzx4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:05.697372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:05.697342    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"
Oct 30 21:34:05.697588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:05.697347    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th" event={"ID":"a3c51fa3-af03-4bc2-9752-251300d7bfa4","Type":"ContainerDied","Data":"bf6c00254fda4c3f5fcc83f5500c17398a7ba8e9df65589c7006779fc19ae829"}
Oct 30 21:34:05.697588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:05.697485    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf6c00254fda4c3f5fcc83f5500c17398a7ba8e9df65589c7006779fc19ae829"
Oct 30 21:34:10.441691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:10.441650    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"]
Oct 30 21:34:10.448144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:10.448115    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-18a043-q2h9d"]
Oct 30 21:34:11.216269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:11.216229    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e071fb1-c954-42fb-9237-ca9923b43d6d" path="/var/lib/kubelet/pods/7e071fb1-c954-42fb-9237-ca9923b43d6d/volumes"
Oct 30 21:34:17.800462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.800426    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"]
Oct 30 21:34:17.801103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.801087    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a3c51fa3-af03-4bc2-9752-251300d7bfa4" containerName="revoke-keys"
Oct 30 21:34:17.801157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.801104    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c51fa3-af03-4bc2-9752-251300d7bfa4" containerName="revoke-keys"
Oct 30 21:34:17.801239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.801229    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c51fa3-af03-4bc2-9752-251300d7bfa4" containerName="revoke-keys"
Oct 30 21:34:17.806483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.806459    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.848938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.848894    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"]
Oct 30 21:34:17.852462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.852438    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:34:17.852820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.852792    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:34:17.853080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.853067    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:34:17.853283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.853269    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:34:17.883268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.878997    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.883268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.879158    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.883268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.879186    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fj9w\" (UniqueName: \"kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.981102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.981068    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.981284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.981186    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.981284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.981213    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fj9w\" (UniqueName: \"kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.981904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.981877    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.983457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.983432    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:17.999522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:17.999498    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fj9w\" (UniqueName: \"kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w\") pod \"maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:18.159288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:18.159206    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:18.493711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:18.493594    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"]
Oct 30 21:34:18.499038 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:34:18.498143    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7697dfb3_a743_45ea_963b_199c9eebb4db.slice/crio-8a5a064e6b734ccac4ce802973a74134513d258a0442ab3c44ad9675ad585415 WatchSource:0}: Error finding container 8a5a064e6b734ccac4ce802973a74134513d258a0442ab3c44ad9675ad585415: Status 404 returned error can't find the container with id 8a5a064e6b734ccac4ce802973a74134513d258a0442ab3c44ad9675ad585415
Oct 30 21:34:18.653171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:18.653137    6695 scope.go:122] "RemoveContainer" containerID="8e393ae32ea200f22a9047f3fa70eac9ed230cf0d8162399af024ff85829946f"
Oct 30 21:34:18.663726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:18.663701    6695 scope.go:122] "RemoveContainer" containerID="64db2fbc031f5c207c7abc466b4209b45797a3c0f313c007bcb7350ec8ebf8f4"
Oct 30 21:34:18.806269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:18.806229    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7" event={"ID":"7697dfb3-a743-45ea-963b-199c9eebb4db","Type":"ContainerStarted","Data":"8a5a064e6b734ccac4ce802973a74134513d258a0442ab3c44ad9675ad585415"}
Oct 30 21:34:19.820541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:19.820504    6695 generic.go:356] "Generic (PLEG): container finished" podID="7697dfb3-a743-45ea-963b-199c9eebb4db" containerID="f524b374afa3ca09ce1abf8f9236e6077ce6786860634f1e7545e055eea8c9a2" exitCode=0
Oct 30 21:34:19.821023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:19.820594    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7" event={"ID":"7697dfb3-a743-45ea-963b-199c9eebb4db","Type":"ContainerDied","Data":"f524b374afa3ca09ce1abf8f9236e6077ce6786860634f1e7545e055eea8c9a2"}
Oct 30 21:34:21.065897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.065253    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:21.132033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.131937    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w\" (UniqueName: \"kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w\") pod \"7697dfb3-a743-45ea-963b-199c9eebb4db\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") "
Oct 30 21:34:21.132033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.132022    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca\") pod \"7697dfb3-a743-45ea-963b-199c9eebb4db\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") "
Oct 30 21:34:21.132281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.132236    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca\") pod \"7697dfb3-a743-45ea-963b-199c9eebb4db\" (UID: \"7697dfb3-a743-45ea-963b-199c9eebb4db\") "
Oct 30 21:34:21.132512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.132480    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca" pod "7697dfb3-a743-45ea-963b-199c9eebb4db" (UID: "7697dfb3-a743-45ea-963b-199c9eebb4db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:21.133056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.133028    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.134531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.134503    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca" pod "7697dfb3-a743-45ea-963b-199c9eebb4db" (UID: "7697dfb3-a743-45ea-963b-199c9eebb4db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:21.135348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.135323    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w" pod "7697dfb3-a743-45ea-963b-199c9eebb4db" (UID: "7697dfb3-a743-45ea-963b-199c9eebb4db"). InnerVolumeSpecName "kube-api-access-6fj9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:21.234316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.234278    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fj9w\" (UniqueName: \"kubernetes.io/projected/7697dfb3-a743-45ea-963b-199c9eebb4db-kube-api-access-6fj9w\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.234316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.234312    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7697dfb3-a743-45ea-963b-199c9eebb4db-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:21.840809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.840780    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"
Oct 30 21:34:21.841013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.840818    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7" event={"ID":"7697dfb3-a743-45ea-963b-199c9eebb4db","Type":"ContainerDied","Data":"8a5a064e6b734ccac4ce802973a74134513d258a0442ab3c44ad9675ad585415"}
Oct 30 21:34:21.841013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:21.840875    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5a064e6b734ccac4ce802973a74134513d258a0442ab3c44ad9675ad585415"
Oct 30 21:34:22.035654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:22.027677    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"]
Oct 30 21:34:22.036042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:22.036011    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b635b941-d548v"]
Oct 30 21:34:23.215735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.215695    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a50edf-5f68-4ef2-9eeb-b60ee63abb18" path="/var/lib/kubelet/pods/45a50edf-5f68-4ef2-9eeb-b60ee63abb18/volumes"
Oct 30 21:34:23.857235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.857144    6695 generic.go:356] "Generic (PLEG): container finished" podID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerID="73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2" exitCode=0
Oct 30 21:34:23.857235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.857215    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" event={"ID":"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba","Type":"ContainerDied","Data":"73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"}
Oct 30 21:34:23.857700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.857687    6695 scope.go:122] "RemoveContainer" containerID="73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"
Oct 30 21:34:23.859325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.859298    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:34:23.859400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.859364    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:34:23.859400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:23.859396    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:34:24.867029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:24.866986    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" event={"ID":"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba","Type":"ContainerStarted","Data":"ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5"}
Oct 30 21:34:25.779232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:25.779198    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:34:25.780739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:25.780709    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:34:25.780878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:25.780772    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:34:26.779822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:26.779786    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:34:26.780253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:26.779851    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:34:27.779043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:27.778997    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:34:27.779226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:27.779060    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:34:28.788660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:28.781647    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:34:28.788660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:28.781709    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:34:29.070555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:29.067271    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"]
Oct 30 21:34:29.103678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:29.103566    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"]
Oct 30 21:34:29.791629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:29.780020    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:34:29.791629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:29.780080    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:34:30.055681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:30.054303    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:34:30.139816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:30.139780    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"]
Oct 30 21:34:30.165539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:30.165487    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-229795-l9h85"]
Oct 30 21:34:31.075170 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:31.075129    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:34:31.093652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:31.093620    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:34:31.095294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:31.095271    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:34:31.223459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:31.223421    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7804c58d-700a-4604-920a-41f91d1f7663" path="/var/lib/kubelet/pods/7804c58d-700a-4604-920a-41f91d1f7663/volumes"
Oct 30 21:34:32.634330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:32.632938    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"]
Oct 30 21:34:32.670423 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:32.670388    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"]
Oct 30 21:34:34.068486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.068418    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr" podUID="337aed11-fa01-4112-aba4-a621ad0bc85b" containerName="payload-processing" containerID="cri-o://f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210" gracePeriod=25
Oct 30 21:34:34.104171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.104118    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm" podUID="a62a5942-dd5c-4c93-891f-61f50f849431" containerName="payload-pre-processing" containerID="cri-o://4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f" gracePeriod=25
Oct 30 21:34:34.394912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.394888    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:34:34.428621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.428578    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:34:34.464123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.464083    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume\") pod \"a62a5942-dd5c-4c93-891f-61f50f849431\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") "
Oct 30 21:34:34.464339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.464322    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj\" (UniqueName: \"kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj\") pod \"a62a5942-dd5c-4c93-891f-61f50f849431\" (UID: \"a62a5942-dd5c-4c93-891f-61f50f849431\") "
Oct 30 21:34:34.464421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.464390    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume\") pod \"337aed11-fa01-4112-aba4-a621ad0bc85b\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") "
Oct 30 21:34:34.464513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.464419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98\" (UniqueName: \"kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98\") pod \"337aed11-fa01-4112-aba4-a621ad0bc85b\" (UID: \"337aed11-fa01-4112-aba4-a621ad0bc85b\") "
Oct 30 21:34:34.464806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.464457    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume" pod "a62a5942-dd5c-4c93-891f-61f50f849431" (UID: "a62a5942-dd5c-4c93-891f-61f50f849431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:34.464965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.464868    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume" pod "337aed11-fa01-4112-aba4-a621ad0bc85b" (UID: "337aed11-fa01-4112-aba4-a621ad0bc85b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:34.465273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.465248    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a62a5942-dd5c-4c93-891f-61f50f849431-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:34.465372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.465278    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/337aed11-fa01-4112-aba4-a621ad0bc85b-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:34.466617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.466585    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98" pod "337aed11-fa01-4112-aba4-a621ad0bc85b" (UID: "337aed11-fa01-4112-aba4-a621ad0bc85b"). InnerVolumeSpecName "kube-api-access-m5z98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:34.467012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.466996    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj" pod "a62a5942-dd5c-4c93-891f-61f50f849431" (UID: "a62a5942-dd5c-4c93-891f-61f50f849431"). InnerVolumeSpecName "kube-api-access-26ssj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:34.566985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.566947    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5z98\" (UniqueName: \"kubernetes.io/projected/337aed11-fa01-4112-aba4-a621ad0bc85b-kube-api-access-m5z98\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:34.566985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:34.566979    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26ssj\" (UniqueName: \"kubernetes.io/projected/a62a5942-dd5c-4c93-891f-61f50f849431-kube-api-access-26ssj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:35.026042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.026004    6695 generic.go:356] "Generic (PLEG): container finished" podID="a62a5942-dd5c-4c93-891f-61f50f849431" containerID="4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f" exitCode=0
Oct 30 21:34:35.026228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.026064    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"
Oct 30 21:34:35.026228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.026088    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm" event={"ID":"a62a5942-dd5c-4c93-891f-61f50f849431","Type":"ContainerDied","Data":"4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f"}
Oct 30 21:34:35.026228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.026147    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm" event={"ID":"a62a5942-dd5c-4c93-891f-61f50f849431","Type":"ContainerDied","Data":"47d03cf5bfd8ad3c70011c79b00310c5ca64cac1cb792c1d2388276b4399296c"}
Oct 30 21:34:35.026228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.026169    6695 scope.go:122] "RemoveContainer" containerID="4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f"
Oct 30 21:34:35.027767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.027743    6695 generic.go:356] "Generic (PLEG): container finished" podID="337aed11-fa01-4112-aba4-a621ad0bc85b" containerID="f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210" exitCode=0
Oct 30 21:34:35.027886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.027789    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"
Oct 30 21:34:35.027886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.027803    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr" event={"ID":"337aed11-fa01-4112-aba4-a621ad0bc85b","Type":"ContainerDied","Data":"f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210"}
Oct 30 21:34:35.027886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.027860    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr" event={"ID":"337aed11-fa01-4112-aba4-a621ad0bc85b","Type":"ContainerDied","Data":"3294a961f84977cfc9610574ce7b4c0bd73f3230090d8fe2e1e73ed126b21476"}
Oct 30 21:34:35.037091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.037074    6695 scope.go:122] "RemoveContainer" containerID="4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f"
Oct 30 21:34:35.037515 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:35.037389    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f\": container with ID starting with 4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f not found: ID does not exist" containerID="4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f"
Oct 30 21:34:35.037515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.037415    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f"} err="failed to get container status \"4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f\": rpc error: code = NotFound desc = could not find container \"4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f\": container with ID starting with 4bfca6de15c9d745fc7e5b24ba222f41611eacbe6112770fb590ca7b3ffc0d9f not found: ID does not exist"
Oct 30 21:34:35.037515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.037431    6695 scope.go:122] "RemoveContainer" containerID="f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210"
Oct 30 21:34:35.047238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.047213    6695 scope.go:122] "RemoveContainer" containerID="f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210"
Oct 30 21:34:35.047516 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:35.047487    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210\": container with ID starting with f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210 not found: ID does not exist" containerID="f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210"
Oct 30 21:34:35.047560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.047525    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210"} err="failed to get container status \"f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210\": rpc error: code = NotFound desc = could not find container \"f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210\": container with ID starting with f4dcae9b0695a458db6d8ff565b3c9259812e9646c95b9192946796786520210 not found: ID does not exist"
Oct 30 21:34:35.077927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.077887    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"]
Oct 30 21:34:35.083627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.083586    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b1659a-67c6695cd9-5nllr"]
Oct 30 21:34:35.115375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.115338    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"]
Oct 30 21:34:35.118953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.118924    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1659a-844c967f75-wm7tm"]
Oct 30 21:34:35.214809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.214776    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337aed11-fa01-4112-aba4-a621ad0bc85b" path="/var/lib/kubelet/pods/337aed11-fa01-4112-aba4-a621ad0bc85b/volumes"
Oct 30 21:34:35.215222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:35.215209    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62a5942-dd5c-4c93-891f-61f50f849431" path="/var/lib/kubelet/pods/a62a5942-dd5c-4c93-891f-61f50f849431/volumes"
Oct 30 21:34:36.765663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:36.765623    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"]
Oct 30 21:34:36.766325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:36.765854    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy" containerID="cri-o://e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba" gracePeriod=30
Oct 30 21:34:37.636024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:37.635978    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92" podUID="0aee02cf-0177-488b-a83a-8571222237ea" containerName="payload-processing" containerID="cri-o://3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d" gracePeriod=25
Oct 30 21:34:37.670996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:37.670949    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb" podUID="372e3a8e-7406-4c82-a782-2da39fb68c74" containerName="payload-pre-processing" containerID="cri-o://65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d" gracePeriod=25
Oct 30 21:34:37.900576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:37.900493    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"]
Oct 30 21:34:37.901997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:37.901730    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6" podUID="419c5cea-64ae-44d8-b67c-624e030c0128" containerName="payload-processing" containerID="cri-o://ba4cfde60c7e1064be4ec74f1fd6e5ac0b28f3e645dd8ffda658fadcc9201528" gracePeriod=30
Oct 30 21:34:38.029821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.029792    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:34:38.045554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.044939    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:34:38.089351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.089310    6695 generic.go:356] "Generic (PLEG): container finished" podID="0aee02cf-0177-488b-a83a-8571222237ea" containerID="3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d" exitCode=0
Oct 30 21:34:38.089564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.089539    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92" event={"ID":"0aee02cf-0177-488b-a83a-8571222237ea","Type":"ContainerDied","Data":"3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d"}
Oct 30 21:34:38.089672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.089618    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92" event={"ID":"0aee02cf-0177-488b-a83a-8571222237ea","Type":"ContainerDied","Data":"950d640adcc03ef960eae64b3a1be5d26ca6e8b525528a4b5c3244c6e059a054"}
Oct 30 21:34:38.089672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.089657    6695 scope.go:122] "RemoveContainer" containerID="3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d"
Oct 30 21:34:38.089813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.089797    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"
Oct 30 21:34:38.093062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.093032    6695 generic.go:356] "Generic (PLEG): container finished" podID="419c5cea-64ae-44d8-b67c-624e030c0128" containerID="ba4cfde60c7e1064be4ec74f1fd6e5ac0b28f3e645dd8ffda658fadcc9201528" exitCode=0
Oct 30 21:34:38.093532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.093217    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6" event={"ID":"419c5cea-64ae-44d8-b67c-624e030c0128","Type":"ContainerDied","Data":"ba4cfde60c7e1064be4ec74f1fd6e5ac0b28f3e645dd8ffda658fadcc9201528"}
Oct 30 21:34:38.095011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.094985    6695 generic.go:356] "Generic (PLEG): container finished" podID="372e3a8e-7406-4c82-a782-2da39fb68c74" containerID="65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d" exitCode=0
Oct 30 21:34:38.095110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.095025    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb" event={"ID":"372e3a8e-7406-4c82-a782-2da39fb68c74","Type":"ContainerDied","Data":"65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d"}
Oct 30 21:34:38.095110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.095064    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb" event={"ID":"372e3a8e-7406-4c82-a782-2da39fb68c74","Type":"ContainerDied","Data":"e3bf9b22fa9f326a060a4215b1863aa687d856c3b73c0e859940573a42cbff78"}
Oct 30 21:34:38.095237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.095137    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"
Oct 30 21:34:38.105511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.105489    6695 scope.go:122] "RemoveContainer" containerID="3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d"
Oct 30 21:34:38.106331 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:38.105854    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d\": container with ID starting with 3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d not found: ID does not exist" containerID="3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d"
Oct 30 21:34:38.106331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.105886    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d"} err="failed to get container status \"3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d\": rpc error: code = NotFound desc = could not find container \"3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d\": container with ID starting with 3b4d19311a1a7c18e93be2724431f4de46e93ca1c91664e7051d23e11791d91d not found: ID does not exist"
Oct 30 21:34:38.106331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.105907    6695 scope.go:122] "RemoveContainer" containerID="65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d"
Oct 30 21:34:38.121686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.121654    6695 scope.go:122] "RemoveContainer" containerID="65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d"
Oct 30 21:34:38.122045 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:38.122024    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d\": container with ID starting with 65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d not found: ID does not exist" containerID="65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d"
Oct 30 21:34:38.122122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.122055    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d"} err="failed to get container status \"65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d\": rpc error: code = NotFound desc = could not find container \"65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d\": container with ID starting with 65cb94fd1882e0ad0535c9a7f990c360d1d79123e80f4e19ded33d402e29d29d not found: ID does not exist"
Oct 30 21:34:38.140171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.140141    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5\" (UniqueName: \"kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5\") pod \"0aee02cf-0177-488b-a83a-8571222237ea\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") "
Oct 30 21:34:38.140325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.140211    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n\" (UniqueName: \"kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n\") pod \"372e3a8e-7406-4c82-a782-2da39fb68c74\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") "
Oct 30 21:34:38.140325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.140279    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume\") pod \"0aee02cf-0177-488b-a83a-8571222237ea\" (UID: \"0aee02cf-0177-488b-a83a-8571222237ea\") "
Oct 30 21:34:38.140474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.140458    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume\") pod \"372e3a8e-7406-4c82-a782-2da39fb68c74\" (UID: \"372e3a8e-7406-4c82-a782-2da39fb68c74\") "
Oct 30 21:34:38.141177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.141138    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume" pod "372e3a8e-7406-4c82-a782-2da39fb68c74" (UID: "372e3a8e-7406-4c82-a782-2da39fb68c74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:38.141351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.141329    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume" pod "0aee02cf-0177-488b-a83a-8571222237ea" (UID: "0aee02cf-0177-488b-a83a-8571222237ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:38.145572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.145541    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n" pod "372e3a8e-7406-4c82-a782-2da39fb68c74" (UID: "372e3a8e-7406-4c82-a782-2da39fb68c74"). InnerVolumeSpecName "kube-api-access-56g4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:38.147957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.147933    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5" pod "0aee02cf-0177-488b-a83a-8571222237ea" (UID: "0aee02cf-0177-488b-a83a-8571222237ea"). InnerVolumeSpecName "kube-api-access-lncc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:38.218754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.218721    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:34:38.256827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.247238    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lncc5\" (UniqueName: \"kubernetes.io/projected/0aee02cf-0177-488b-a83a-8571222237ea-kube-api-access-lncc5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.256827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.247279    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56g4n\" (UniqueName: \"kubernetes.io/projected/372e3a8e-7406-4c82-a782-2da39fb68c74-kube-api-access-56g4n\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.256827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.247294    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee02cf-0177-488b-a83a-8571222237ea-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.256827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.247308    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372e3a8e-7406-4c82-a782-2da39fb68c74-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.354291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.351156    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume\") pod \"419c5cea-64ae-44d8-b67c-624e030c0128\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") "
Oct 30 21:34:38.354291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.351275    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw\" (UniqueName: \"kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw\") pod \"419c5cea-64ae-44d8-b67c-624e030c0128\" (UID: \"419c5cea-64ae-44d8-b67c-624e030c0128\") "
Oct 30 21:34:38.354291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.352687    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume" pod "419c5cea-64ae-44d8-b67c-624e030c0128" (UID: "419c5cea-64ae-44d8-b67c-624e030c0128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:38.359813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.359759    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw" pod "419c5cea-64ae-44d8-b67c-624e030c0128" (UID: "419c5cea-64ae-44d8-b67c-624e030c0128"). InnerVolumeSpecName "kube-api-access-4pxmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:38.505115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.505023    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419c5cea-64ae-44d8-b67c-624e030c0128-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.505115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.505064    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pxmw\" (UniqueName: \"kubernetes.io/projected/419c5cea-64ae-44d8-b67c-624e030c0128-kube-api-access-4pxmw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.566186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.566152    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"]
Oct 30 21:34:38.570391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.570365    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-85c866d979-qd7vb"]
Oct 30 21:34:38.588798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.588765    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"]
Oct 30 21:34:38.592252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.592207    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-5dc7f669c7-7lm92"]
Oct 30 21:34:38.664150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.657303    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"]
Oct 30 21:34:38.664150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:38.657405    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c027ae-vr9th"]
Oct 30 21:34:39.105019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.104989    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"
Oct 30 21:34:39.105019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.105010    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6" event={"ID":"419c5cea-64ae-44d8-b67c-624e030c0128","Type":"ContainerDied","Data":"149c82a1a7c47be2c20bfeb8e704225c1303f8ddc2516970d7719968439893a1"}
Oct 30 21:34:39.105583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.105105    6695 scope.go:122] "RemoveContainer" containerID="ba4cfde60c7e1064be4ec74f1fd6e5ac0b28f3e645dd8ffda658fadcc9201528"
Oct 30 21:34:39.180724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.180696    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"]
Oct 30 21:34:39.200370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.200337    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-97c98bf9d-n6sz6"]
Oct 30 21:34:39.237297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.237258    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aee02cf-0177-488b-a83a-8571222237ea" path="/var/lib/kubelet/pods/0aee02cf-0177-488b-a83a-8571222237ea/volumes"
Oct 30 21:34:39.237883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.237862    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372e3a8e-7406-4c82-a782-2da39fb68c74" path="/var/lib/kubelet/pods/372e3a8e-7406-4c82-a782-2da39fb68c74/volumes"
Oct 30 21:34:39.238425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.238405    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419c5cea-64ae-44d8-b67c-624e030c0128" path="/var/lib/kubelet/pods/419c5cea-64ae-44d8-b67c-624e030c0128/volumes"
Oct 30 21:34:39.239443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:39.239420    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c51fa3-af03-4bc2-9752-251300d7bfa4" path="/var/lib/kubelet/pods/a3c51fa3-af03-4bc2-9752-251300d7bfa4/volumes"
Oct 30 21:34:42.085159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.085120    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"]
Oct 30 21:34:42.092033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.092008    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:34:42.151562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.151530    6695 generic.go:356] "Generic (PLEG): container finished" podID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerID="e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba" exitCode=0
Oct 30 21:34:42.151801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.151577    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"
Oct 30 21:34:42.151801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.151632    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" event={"ID":"a83a9b10-e409-4d52-b108-41fe7289ea5f","Type":"ContainerDied","Data":"e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba"}
Oct 30 21:34:42.151801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.151686    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc" event={"ID":"a83a9b10-e409-4d52-b108-41fe7289ea5f","Type":"ContainerDied","Data":"7025f94bf2d4e4bf6881d7a26c2badeb7b57284a7935910bbde558fca39e7531"}
Oct 30 21:34:42.151801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.151714    6695 scope.go:122] "RemoveContainer" containerID="e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba"
Oct 30 21:34:42.161997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.161974    6695 scope.go:122] "RemoveContainer" containerID="63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"
Oct 30 21:34:42.172191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.172169    6695 scope.go:122] "RemoveContainer" containerID="e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba"
Oct 30 21:34:42.172444 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:42.172425    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba\": container with ID starting with e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba not found: ID does not exist" containerID="e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba"
Oct 30 21:34:42.172528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.172453    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba"} err="failed to get container status \"e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba\": rpc error: code = NotFound desc = could not find container \"e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba\": container with ID starting with e1ec090b81c900f211dff1fc06758046459eed1c12b28f95fea70006f1812fba not found: ID does not exist"
Oct 30 21:34:42.172528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.172479    6695 scope.go:122] "RemoveContainer" containerID="63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"
Oct 30 21:34:42.172808 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:42.172765    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b\": container with ID starting with 63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b not found: ID does not exist" containerID="63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"
Oct 30 21:34:42.172905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.172804    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b"} err="failed to get container status \"63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b\": rpc error: code = NotFound desc = could not find container \"63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b\": container with ID starting with 63c3083f8c54d0ed4e690a406341b8fd95aa3d4a0a91917c6c40908b50cf084b not found: ID does not exist"
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185089    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185167    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185213    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185243    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185279    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185344    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185380    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185561    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.185654    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk\") pod \"a83a9b10-e409-4d52-b108-41fe7289ea5f\" (UID: \"a83a9b10-e409-4d52-b108-41fe7289ea5f\") "
Oct 30 21:34:42.188968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.187431    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:42.189671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.189138    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:42.189671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.189303    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:42.197681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.189889    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.197681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.189913    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.197681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.189929    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.197681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.190853    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:42.197681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.192066    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:42.203290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.203219    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "kube-api-access-s2sjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:42.206754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.205157    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:42.245893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.242776    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:42.246629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.246286    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy" pod "a83a9b10-e409-4d52-b108-41fe7289ea5f" (UID: "a83a9b10-e409-4d52-b108-41fe7289ea5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:42.291800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.291758    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.291976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.291875    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83a9b10-e409-4d52-b108-41fe7289ea5f-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.291976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.291899    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2sjk\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-kube-api-access-s2sjk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.291976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.291914    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.291976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.291926    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83a9b10-e409-4d52-b108-41fe7289ea5f-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.291976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.291938    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83a9b10-e409-4d52-b108-41fe7289ea5f-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.457865    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"]
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465624    6695 cpu_manager.go:414] "containerMap: removing container" podUID="372e3a8e-7406-4c82-a782-2da39fb68c74" containerName="payload-pre-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465653    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372e3a8e-7406-4c82-a782-2da39fb68c74" containerName="payload-pre-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465680    6695 cpu_manager.go:414] "containerMap: removing container" podUID="419c5cea-64ae-44d8-b67c-624e030c0128" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465688    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419c5cea-64ae-44d8-b67c-624e030c0128" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465698    6695 cpu_manager.go:414] "containerMap: removing container" podUID="337aed11-fa01-4112-aba4-a621ad0bc85b" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465707    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337aed11-fa01-4112-aba4-a621ad0bc85b" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465719    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0aee02cf-0177-488b-a83a-8571222237ea" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465726    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aee02cf-0177-488b-a83a-8571222237ea" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465748    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465755    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465768    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a62a5942-dd5c-4c93-891f-61f50f849431" containerName="payload-pre-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465776    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62a5942-dd5c-4c93-891f-61f50f849431" containerName="payload-pre-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465812    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7697dfb3-a743-45ea-963b-199c9eebb4db" containerName="revoke-keys"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465819    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7697dfb3-a743-45ea-963b-199c9eebb4db" containerName="revoke-keys"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.465989    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7697dfb3-a743-45ea-963b-199c9eebb4db" containerName="revoke-keys"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.466009    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="372e3a8e-7406-4c82-a782-2da39fb68c74" containerName="payload-pre-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.466019    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="419c5cea-64ae-44d8-b67c-624e030c0128" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.466029    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="337aed11-fa01-4112-aba4-a621ad0bc85b" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.466038    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aee02cf-0177-488b-a83a-8571222237ea" containerName="payload-processing"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.466047    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy"
Oct 30 21:34:42.468011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.466055    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62a5942-dd5c-4c93-891f-61f50f849431" containerName="payload-pre-processing"
Oct 30 21:34:42.541645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.506921    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.541645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.510220    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"]
Oct 30 21:34:42.575640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.561860    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:34:42.575640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.562148    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:34:42.575640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.562260    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:34:42.575640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.562897    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:34:42.622686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.616968    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.622686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.617093    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgpf\" (UniqueName: \"kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.622686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.617168    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.644723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.644678    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"]
Oct 30 21:34:42.651403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.651368    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-229795-openshift-default-cd8bcbc59-vg2gc"]
Oct 30 21:34:42.738543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.737554    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgpf\" (UniqueName: \"kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.738543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.737676    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.738543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.737945    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.746626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.741229    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.758671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.747386    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.760254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.760191    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgpf\" (UniqueName: \"kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf\") pod \"maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:42.893138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:42.892847    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:43.240723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:43.240687    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" path="/var/lib/kubelet/pods/a83a9b10-e409-4d52-b108-41fe7289ea5f/volumes"
Oct 30 21:34:43.338675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:43.338336    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"]
Oct 30 21:34:43.341050 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:34:43.341022    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e2fcbc_85f5_4ced_9725_38e235a6bfb7.slice/crio-8e953ec1a0c959d9156374116badbc04ecb04710037ca229bc7788e4d8bbdb39 WatchSource:0}: Error finding container 8e953ec1a0c959d9156374116badbc04ecb04710037ca229bc7788e4d8bbdb39: Status 404 returned error can't find the container with id 8e953ec1a0c959d9156374116badbc04ecb04710037ca229bc7788e4d8bbdb39
Oct 30 21:34:44.171975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:44.171876    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g" event={"ID":"60e2fcbc-85f5-4ced-9725-38e235a6bfb7","Type":"ContainerStarted","Data":"3b37fcd799fdd2eb4e5cd1a0d92cc20e8df962c055a08cfa767da818a0ef30d0"}
Oct 30 21:34:44.171975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:44.171948    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g" event={"ID":"60e2fcbc-85f5-4ced-9725-38e235a6bfb7","Type":"ContainerStarted","Data":"8e953ec1a0c959d9156374116badbc04ecb04710037ca229bc7788e4d8bbdb39"}
Oct 30 21:34:44.209809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:44.209762    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g" podStartSLOduration=2.209747786 podStartE2EDuration="2.209747786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:44.20868816 +0000 UTC m=+1467.131146452" watchObservedRunningTime="2026-10-30 21:34:44.209747786 +0000 UTC m=+1467.132206164"
Oct 30 21:34:45.182188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.182141    6695 generic.go:356] "Generic (PLEG): container finished" podID="60e2fcbc-85f5-4ced-9725-38e235a6bfb7" containerID="3b37fcd799fdd2eb4e5cd1a0d92cc20e8df962c055a08cfa767da818a0ef30d0" exitCode=0
Oct 30 21:34:45.182736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.182287    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g" event={"ID":"60e2fcbc-85f5-4ced-9725-38e235a6bfb7","Type":"ContainerDied","Data":"3b37fcd799fdd2eb4e5cd1a0d92cc20e8df962c055a08cfa767da818a0ef30d0"}
Oct 30 21:34:45.286373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.286325    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"]
Oct 30 21:34:45.286636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.286589    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" containerID="cri-o://018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e" gracePeriod=30
Oct 30 21:34:45.838215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.838177    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"]
Oct 30 21:34:45.838866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.838851    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy"
Oct 30 21:34:45.838946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.838868    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy"
Oct 30 21:34:45.839005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.838968    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83a9b10-e409-4d52-b108-41fe7289ea5f" containerName="istio-proxy"
Oct 30 21:34:45.842759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.842737    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:45.860368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.860278    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"]
Oct 30 21:34:45.904648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.903294    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxcd\" (UniqueName: \"kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:45.904648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.903382    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:45.904648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:45.903411    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.005493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.005451    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.005493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.005499    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.005826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.005807    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxcd\" (UniqueName: \"kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.006806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.006777    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.008486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.008463    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.024442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.024403    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxcd\" (UniqueName: \"kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd\") pod \"maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.157188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.157110    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:46.395199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.395172    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:46.474969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.474895    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"]
Oct 30 21:34:46.481909 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:34:46.481867    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad392f41_22a5_4f22_8057_9320f3002821.slice/crio-3956dff1ac3311e11dbea6386267ebef96838e257abe8de7d4d40165dcfbe40f WatchSource:0}: Error finding container 3956dff1ac3311e11dbea6386267ebef96838e257abe8de7d4d40165dcfbe40f: Status 404 returned error can't find the container with id 3956dff1ac3311e11dbea6386267ebef96838e257abe8de7d4d40165dcfbe40f
Oct 30 21:34:46.513247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.513215    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf\" (UniqueName: \"kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf\") pod \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") "
Oct 30 21:34:46.513396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.513270    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca\") pod \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") "
Oct 30 21:34:46.513396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.513353    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca\") pod \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\" (UID: \"60e2fcbc-85f5-4ced-9725-38e235a6bfb7\") "
Oct 30 21:34:46.513758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.513704    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca" pod "60e2fcbc-85f5-4ced-9725-38e235a6bfb7" (UID: "60e2fcbc-85f5-4ced-9725-38e235a6bfb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:46.514179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.514156    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:46.515867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.515839    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf" pod "60e2fcbc-85f5-4ced-9725-38e235a6bfb7" (UID: "60e2fcbc-85f5-4ced-9725-38e235a6bfb7"). InnerVolumeSpecName "kube-api-access-5zgpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:46.515976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.515933    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca" pod "60e2fcbc-85f5-4ced-9725-38e235a6bfb7" (UID: "60e2fcbc-85f5-4ced-9725-38e235a6bfb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:46.615392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.615353    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zgpf\" (UniqueName: \"kubernetes.io/projected/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-kube-api-access-5zgpf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:46.615392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:46.615390    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e2fcbc-85f5-4ced-9725-38e235a6bfb7-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:47.086439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.086394    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j" podUID="e14e2126-7387-4d5d-9aa4-69f31f8c0c57" containerName="payload-processing" containerID="cri-o://94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e" gracePeriod=25
Oct 30 21:34:47.318687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.308032    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc" event={"ID":"ad392f41-22a5-4f22-8057-9320f3002821","Type":"ContainerStarted","Data":"4facad9d92336adf9ecf108c4dbbd6a200bd48bf85940c0e0417ef43af62727c"}
Oct 30 21:34:47.318687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.308104    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc" event={"ID":"ad392f41-22a5-4f22-8057-9320f3002821","Type":"ContainerStarted","Data":"3956dff1ac3311e11dbea6386267ebef96838e257abe8de7d4d40165dcfbe40f"}
Oct 30 21:34:47.318687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.316433    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g" event={"ID":"60e2fcbc-85f5-4ced-9725-38e235a6bfb7","Type":"ContainerDied","Data":"8e953ec1a0c959d9156374116badbc04ecb04710037ca229bc7788e4d8bbdb39"}
Oct 30 21:34:47.318687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.316489    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e953ec1a0c959d9156374116badbc04ecb04710037ca229bc7788e4d8bbdb39"
Oct 30 21:34:47.318687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.316559    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"
Oct 30 21:34:47.350844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.350778    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc" podStartSLOduration=2.350757844 podStartE2EDuration="2.350757844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:47.347381122 +0000 UTC m=+1470.269839421" watchObservedRunningTime="2026-10-30 21:34:47.350757844 +0000 UTC m=+1470.273216143"
Oct 30 21:34:47.521826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.521480    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:34:47.653478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.653445    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume\") pod \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") "
Oct 30 21:34:47.653710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.653686    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp\" (UniqueName: \"kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp\") pod \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\" (UID: \"e14e2126-7387-4d5d-9aa4-69f31f8c0c57\") "
Oct 30 21:34:47.654210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.654186    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume" pod "e14e2126-7387-4d5d-9aa4-69f31f8c0c57" (UID: "e14e2126-7387-4d5d-9aa4-69f31f8c0c57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:47.654324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.654309    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:47.655987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.655958    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp" pod "e14e2126-7387-4d5d-9aa4-69f31f8c0c57" (UID: "e14e2126-7387-4d5d-9aa4-69f31f8c0c57"). InnerVolumeSpecName "kube-api-access-jscnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:47.755870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:47.755827    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jscnp\" (UniqueName: \"kubernetes.io/projected/e14e2126-7387-4d5d-9aa4-69f31f8c0c57-kube-api-access-jscnp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:48.325048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.325011    6695 generic.go:356] "Generic (PLEG): container finished" podID="e14e2126-7387-4d5d-9aa4-69f31f8c0c57" containerID="94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e" exitCode=0
Oct 30 21:34:48.327238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.325059    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"
Oct 30 21:34:48.327238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.325103    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j" event={"ID":"e14e2126-7387-4d5d-9aa4-69f31f8c0c57","Type":"ContainerDied","Data":"94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e"}
Oct 30 21:34:48.327238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.325163    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j" event={"ID":"e14e2126-7387-4d5d-9aa4-69f31f8c0c57","Type":"ContainerDied","Data":"62cef96d7c2cdf56e8a6a91083bb63f1656cde2d3e5d7a367f5d321d70563b3e"}
Oct 30 21:34:48.327238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.325195    6695 scope.go:122] "RemoveContainer" containerID="94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e"
Oct 30 21:34:48.327238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.326976    6695 generic.go:356] "Generic (PLEG): container finished" podID="ad392f41-22a5-4f22-8057-9320f3002821" containerID="4facad9d92336adf9ecf108c4dbbd6a200bd48bf85940c0e0417ef43af62727c" exitCode=0
Oct 30 21:34:48.327238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.327062    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc" event={"ID":"ad392f41-22a5-4f22-8057-9320f3002821","Type":"ContainerDied","Data":"4facad9d92336adf9ecf108c4dbbd6a200bd48bf85940c0e0417ef43af62727c"}
Oct 30 21:34:48.336588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.336566    6695 scope.go:122] "RemoveContainer" containerID="94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e"
Oct 30 21:34:48.336925 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:48.336904    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e\": container with ID starting with 94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e not found: ID does not exist" containerID="94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e"
Oct 30 21:34:48.336979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.336934    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e"} err="failed to get container status \"94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e\": rpc error: code = NotFound desc = could not find container \"94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e\": container with ID starting with 94e8ae746ac0dbae34fb9c5c71da5c0e3f2a2ac9b6575f021d02148bf814c48e not found: ID does not exist"
Oct 30 21:34:48.390504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.390475    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"]
Oct 30 21:34:48.405551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:48.405523    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-6bf78c5b8c-rp26j"]
Oct 30 21:34:49.215004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.214969    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14e2126-7387-4d5d-9aa4-69f31f8c0c57" path="/var/lib/kubelet/pods/e14e2126-7387-4d5d-9aa4-69f31f8c0c57/volumes"
Oct 30 21:34:49.335282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.335248    6695 generic.go:356] "Generic (PLEG): container finished" podID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerID="8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440" exitCode=0
Oct 30 21:34:49.335473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.335309    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" event={"ID":"7c1730f8-e0e3-4345-9952-cde0a41a63b5","Type":"ContainerDied","Data":"8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"}
Oct 30 21:34:49.336382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.336352    6695 scope.go:122] "RemoveContainer" containerID="8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"
Oct 30 21:34:49.346076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.346036    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:34:49.346211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.346115    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:34:49.346211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.346160    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:34:49.571679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.571648    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:49.682968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.682936    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca\") pod \"ad392f41-22a5-4f22-8057-9320f3002821\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") "
Oct 30 21:34:49.683145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.683120    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd\" (UniqueName: \"kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd\") pod \"ad392f41-22a5-4f22-8057-9320f3002821\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") "
Oct 30 21:34:49.683205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.683187    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca\") pod \"ad392f41-22a5-4f22-8057-9320f3002821\" (UID: \"ad392f41-22a5-4f22-8057-9320f3002821\") "
Oct 30 21:34:49.683386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.683357    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca" pod "ad392f41-22a5-4f22-8057-9320f3002821" (UID: "ad392f41-22a5-4f22-8057-9320f3002821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:49.683815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.683780    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:49.685287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.685252    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd" pod "ad392f41-22a5-4f22-8057-9320f3002821" (UID: "ad392f41-22a5-4f22-8057-9320f3002821"). InnerVolumeSpecName "kube-api-access-bvxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:49.685396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.685377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca" pod "ad392f41-22a5-4f22-8057-9320f3002821" (UID: "ad392f41-22a5-4f22-8057-9320f3002821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:49.754644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.754582    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:34:49.754808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.754666    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:34:49.784998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.784965    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad392f41-22a5-4f22-8057-9320f3002821-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:49.784998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:49.784997    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxcd\" (UniqueName: \"kubernetes.io/projected/ad392f41-22a5-4f22-8057-9320f3002821-kube-api-access-bvxcd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.355678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.355648    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"
Oct 30 21:34:50.356156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.355780    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc" event={"ID":"ad392f41-22a5-4f22-8057-9320f3002821","Type":"ContainerDied","Data":"3956dff1ac3311e11dbea6386267ebef96838e257abe8de7d4d40165dcfbe40f"}
Oct 30 21:34:50.356156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.355838    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3956dff1ac3311e11dbea6386267ebef96838e257abe8de7d4d40165dcfbe40f"
Oct 30 21:34:50.358242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.358216    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" event={"ID":"7c1730f8-e0e3-4345-9952-cde0a41a63b5","Type":"ContainerStarted","Data":"2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08"}
Oct 30 21:34:50.403631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.403569    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:34:50.405053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.404929    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 21:34:50.405053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.405003    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 21:34:50.678571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.678542    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:34:50.812627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812387    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.812627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812565    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.812627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812594    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.812951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812668    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.812951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812698    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.812951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812916    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.813112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812959    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.813112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.812991    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.813112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.813033    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs\") pod \"8002939f-fa35-4c34-bbfd-da44de646313\" (UID: \"8002939f-fa35-4c34-bbfd-da44de646313\") "
Oct 30 21:34:50.859625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.842754    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:50.882625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.866993    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:50.882625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.868326    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:50.893629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.890252    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:50.893629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.890462    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:50.894749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.894675    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "kube-api-access-tszdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:50.895764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.895735    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:50.896197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.896167    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:50.896581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.896529    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy" pod "8002939f-fa35-4c34-bbfd-da44de646313" (UID: "8002939f-fa35-4c34-bbfd-da44de646313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937745    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002939f-fa35-4c34-bbfd-da44de646313-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937786    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937802    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937818    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002939f-fa35-4c34-bbfd-da44de646313-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937833    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tszdg\" (UniqueName: \"kubernetes.io/projected/8002939f-fa35-4c34-bbfd-da44de646313-kube-api-access-tszdg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937849    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937864    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937877    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:50.938832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:50.937891    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002939f-fa35-4c34-bbfd-da44de646313-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:51.367556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.367518    6695 generic.go:356] "Generic (PLEG): container finished" podID="8002939f-fa35-4c34-bbfd-da44de646313" containerID="018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e" exitCode=0
Oct 30 21:34:51.368081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.367563    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" event={"ID":"8002939f-fa35-4c34-bbfd-da44de646313","Type":"ContainerDied","Data":"018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e"}
Oct 30 21:34:51.368081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.367572    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"
Oct 30 21:34:51.368081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.367647    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf" event={"ID":"8002939f-fa35-4c34-bbfd-da44de646313","Type":"ContainerDied","Data":"343b9c5a707363c299b457207b215d24cfaf18aea44075d8e9d8e70501df34cd"}
Oct 30 21:34:51.368081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.367672    6695 scope.go:122] "RemoveContainer" containerID="018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e"
Oct 30 21:34:51.378175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.378156    6695 scope.go:122] "RemoveContainer" containerID="4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"
Oct 30 21:34:51.388244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.388223    6695 scope.go:122] "RemoveContainer" containerID="018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e"
Oct 30 21:34:51.388524 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:51.388504    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e\": container with ID starting with 018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e not found: ID does not exist" containerID="018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e"
Oct 30 21:34:51.388581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.388536    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e"} err="failed to get container status \"018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e\": rpc error: code = NotFound desc = could not find container \"018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e\": container with ID starting with 018ca438a8432d40bb5c588e493ae0379e44060203b0cdfd69fc1b914f47bc9e not found: ID does not exist"
Oct 30 21:34:51.388581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.388555    6695 scope.go:122] "RemoveContainer" containerID="4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"
Oct 30 21:34:51.388858 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:34:51.388840    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d\": container with ID starting with 4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d not found: ID does not exist" containerID="4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"
Oct 30 21:34:51.388906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.388866    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d"} err="failed to get container status \"4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d\": rpc error: code = NotFound desc = could not find container \"4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d\": container with ID starting with 4f43dd515893d2e4844158200249540d8a9a9e006d47538b9ee771ef7fa4fb8d not found: ID does not exist"
Oct 30 21:34:51.404275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.404248    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 21:34:51.404371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.404321    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 21:34:51.440779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.440744    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"]
Oct 30 21:34:51.450851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:51.450819    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c027ae-openshift-default-6ffd9b8f55-vnnrf"]
Oct 30 21:34:52.411630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.411513    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 21:34:52.411630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.411590    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 21:34:52.488628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.484007    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"]
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496323    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496353    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496466    6695 cpu_manager.go:414] "containerMap: removing container" podUID="60e2fcbc-85f5-4ced-9725-38e235a6bfb7" containerName="revoke-keys"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496475    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e2fcbc-85f5-4ced-9725-38e235a6bfb7" containerName="revoke-keys"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496488    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ad392f41-22a5-4f22-8057-9320f3002821" containerName="revoke-keys"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496496    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad392f41-22a5-4f22-8057-9320f3002821" containerName="revoke-keys"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496511    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e14e2126-7387-4d5d-9aa4-69f31f8c0c57" containerName="payload-processing"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496519    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14e2126-7387-4d5d-9aa4-69f31f8c0c57" containerName="payload-processing"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496692    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e2fcbc-85f5-4ced-9725-38e235a6bfb7" containerName="revoke-keys"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496706    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad392f41-22a5-4f22-8057-9320f3002821" containerName="revoke-keys"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496716    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14e2126-7387-4d5d-9aa4-69f31f8c0c57" containerName="payload-processing"
Oct 30 21:34:52.502627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.496728    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy"
Oct 30 21:34:52.562628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.545863    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"]
Oct 30 21:34:52.562628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.545995    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.598254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.568063    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-38e5bf-dockercfg-cg9dd\""
Oct 30 21:34:52.599160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.580756    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-38e5bf\""
Oct 30 21:34:52.657598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.655700    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rjn\" (UniqueName: \"kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn\") pod \"payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.657598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.655956    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.659678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.658439    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"]
Oct 30 21:34:52.659830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.659744    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy"
Oct 30 21:34:52.659830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.659763    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy"
Oct 30 21:34:52.660129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.660107    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8002939f-fa35-4c34-bbfd-da44de646313" containerName="istio-proxy"
Oct 30 21:34:52.666814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.665112    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.670775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.670174    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"]
Oct 30 21:34:52.757584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.757542    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.757828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.757691    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rjn\" (UniqueName: \"kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn\") pod \"payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.757828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.757725    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.757828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.757798    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnzn\" (UniqueName: \"kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.758383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.758356    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.777628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.777565    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rjn\" (UniqueName: \"kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn\") pod \"payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.859381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.859323    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.859584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.859415    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnzn\" (UniqueName: \"kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.860139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.860118    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.878106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.878076    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnzn\" (UniqueName: \"kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn\") pod \"payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:52.966403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.966311    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:52.989252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:52.988664    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:53.239586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:53.239507    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8002939f-fa35-4c34-bbfd-da44de646313" path="/var/lib/kubelet/pods/8002939f-fa35-4c34-bbfd-da44de646313/volumes"
Oct 30 21:34:53.375522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:53.374938    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"]
Oct 30 21:34:53.385686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:53.381942    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"]
Oct 30 21:34:53.405174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:53.405008    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 21:34:53.405174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:53.405101    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 21:34:53.450213 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:34:53.450178    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eef253_0b20_4a8f_827b_c7777bca0bb3.slice/crio-829a85843e6e646e7870c91b641b91bf1db60541173dd701c345913468155ab9 WatchSource:0}: Error finding container 829a85843e6e646e7870c91b641b91bf1db60541173dd701c345913468155ab9: Status 404 returned error can't find the container with id 829a85843e6e646e7870c91b641b91bf1db60541173dd701c345913468155ab9
Oct 30 21:34:53.455219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:53.455185    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79" event={"ID":"bd2d7192-a9d1-4154-a69d-088b5ef30843","Type":"ContainerStarted","Data":"289108f303dbd34586f657c3a576ef566a80bac681d38fe13a873d5968f69950"}
Oct 30 21:34:54.347186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.347153    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:34:54.403907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.403874    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 21:34:54.404109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.403930    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 21:34:54.464005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.463964    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79" event={"ID":"bd2d7192-a9d1-4154-a69d-088b5ef30843","Type":"ContainerStarted","Data":"c0e3af2c7a9cede1ee59491bc243a9437e3eac64a3fa7cdf40af7e2c1e5dac30"}
Oct 30 21:34:54.464466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.464056    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:34:54.465594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.465570    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6" event={"ID":"a6eef253-0b20-4a8f-827b-c7777bca0bb3","Type":"ContainerStarted","Data":"7e0b777b64074924dadd5bb3b9a95fd9484dab3e3c6dbaea28d5ddec4845ffdd"}
Oct 30 21:34:54.465761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.465642    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:34:54.465761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.465666    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6" event={"ID":"a6eef253-0b20-4a8f-827b-c7777bca0bb3","Type":"ContainerStarted","Data":"829a85843e6e646e7870c91b641b91bf1db60541173dd701c345913468155ab9"}
Oct 30 21:34:54.503051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.502971    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79" podStartSLOduration=2.502956808 podStartE2EDuration="2.502956808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:54.501009026 +0000 UTC m=+1477.423467403" watchObservedRunningTime="2026-10-30 21:34:54.502956808 +0000 UTC m=+1477.425415123"
Oct 30 21:34:54.557545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.557488    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6" podStartSLOduration=2.557469915 podStartE2EDuration="2.557469915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:54.553771446 +0000 UTC m=+1477.476229744" watchObservedRunningTime="2026-10-30 21:34:54.557469915 +0000 UTC m=+1477.479928213"
Oct 30 21:34:54.957048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.957012    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"]
Oct 30 21:34:54.957506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:54.957476    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r" podUID="69bdea97-70db-461b-9e7b-16e382927942" containerName="payload-pre-processing" containerID="cri-o://4ca0522d4fab920be357e744ef85a1c89f98d3d0f20f22e2b7ae13c10d1867f3" gracePeriod=30
Oct 30 21:34:55.432307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.410916    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 21:34:55.432307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.410997    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 21:34:55.483187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.482115    6695 generic.go:356] "Generic (PLEG): container finished" podID="69bdea97-70db-461b-9e7b-16e382927942" containerID="4ca0522d4fab920be357e744ef85a1c89f98d3d0f20f22e2b7ae13c10d1867f3" exitCode=0
Oct 30 21:34:55.483187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.482651    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r" event={"ID":"69bdea97-70db-461b-9e7b-16e382927942","Type":"ContainerDied","Data":"4ca0522d4fab920be357e744ef85a1c89f98d3d0f20f22e2b7ae13c10d1867f3"}
Oct 30 21:34:55.516626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.509682    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:34:55.667462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.667391    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"]
Oct 30 21:34:55.673556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.673326    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume\") pod \"69bdea97-70db-461b-9e7b-16e382927942\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") "
Oct 30 21:34:55.673556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.673556    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw\" (UniqueName: \"kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw\") pod \"69bdea97-70db-461b-9e7b-16e382927942\" (UID: \"69bdea97-70db-461b-9e7b-16e382927942\") "
Oct 30 21:34:55.675357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.675322    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume" pod "69bdea97-70db-461b-9e7b-16e382927942" (UID: "69bdea97-70db-461b-9e7b-16e382927942"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:55.688205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.688109    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7222e2-9fv2g"]
Oct 30 21:34:55.705713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.705669    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw" pod "69bdea97-70db-461b-9e7b-16e382927942" (UID: "69bdea97-70db-461b-9e7b-16e382927942"). InnerVolumeSpecName "kube-api-access-n2vjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:55.788643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.788585    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bdea97-70db-461b-9e7b-16e382927942-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:55.788643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:55.788648    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vjw\" (UniqueName: \"kubernetes.io/projected/69bdea97-70db-461b-9e7b-16e382927942-kube-api-access-n2vjw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:34:56.409402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.409371    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:34:56.410392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.410370    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:34:56.502384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.502349    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"
Oct 30 21:34:56.502384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.502370    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r" event={"ID":"69bdea97-70db-461b-9e7b-16e382927942","Type":"ContainerDied","Data":"e3bcd8f33c16dabfb1b8a06e96f92988693a06a8553345f4d7e492cec29d5c98"}
Oct 30 21:34:56.502972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.502446    6695 scope.go:122] "RemoveContainer" containerID="4ca0522d4fab920be357e744ef85a1c89f98d3d0f20f22e2b7ae13c10d1867f3"
Oct 30 21:34:56.556844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.556791    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"]
Oct 30 21:34:56.561572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.561538    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7222e2-5799b774b7-q798r"]
Oct 30 21:34:56.876021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.875964    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"]
Oct 30 21:34:56.876252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.876227    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy" containerID="cri-o://7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65" gracePeriod=30
Oct 30 21:34:56.879158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.879121    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"]
Oct 30 21:34:56.879375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.879349    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy" containerID="cri-o://5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301" gracePeriod=30
Oct 30 21:34:56.886154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.886127    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"]
Oct 30 21:34:56.886397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:56.886375    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy" containerID="cri-o://b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22" gracePeriod=30
Oct 30 21:34:57.214395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:57.214319    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e2fcbc-85f5-4ced-9725-38e235a6bfb7" path="/var/lib/kubelet/pods/60e2fcbc-85f5-4ced-9725-38e235a6bfb7/volumes"
Oct 30 21:34:57.214797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:34:57.214779    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bdea97-70db-461b-9e7b-16e382927942" path="/var/lib/kubelet/pods/69bdea97-70db-461b-9e7b-16e382927942/volumes"
Oct 30 21:35:01.139849 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:01.139805    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:35:01.401577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.401471    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"]
Oct 30 21:35:01.402665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.402643    6695 cpu_manager.go:414] "containerMap: removing container" podUID="69bdea97-70db-461b-9e7b-16e382927942" containerName="payload-pre-processing"
Oct 30 21:35:01.402817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.402804    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bdea97-70db-461b-9e7b-16e382927942" containerName="payload-pre-processing"
Oct 30 21:35:01.403121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.403091    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bdea97-70db-461b-9e7b-16e382927942" containerName="payload-pre-processing"
Oct 30 21:35:01.415973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.415580    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.428646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.426717    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"]
Oct 30 21:35:01.436902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.436865    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"]
Oct 30 21:35:01.441294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.441269    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.442238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.442220    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0e991\""
Oct 30 21:35:01.442668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.442223    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b0e991-dockercfg-9lvrw\""
Oct 30 21:35:01.468259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.468006    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"]
Oct 30 21:35:01.500529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.500479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ccs\" (UniqueName: \"kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs\") pod \"payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.500765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.500687    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.500765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.500745    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.500956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.500933    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8tk\" (UniqueName: \"kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.602445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.602410    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8tk\" (UniqueName: \"kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.602622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.602561    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ccs\" (UniqueName: \"kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs\") pod \"payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.602622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.602615    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.602731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.602647    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.603652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.603591    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.603713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.603636    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.622224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.622181    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8tk\" (UniqueName: \"kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.622407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.622197    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ccs\" (UniqueName: \"kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs\") pod \"payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:01.740314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.740229    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:01.757932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:01.757897    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:02.055727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.054488    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"]
Oct 30 21:35:02.318384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.318355    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"]
Oct 30 21:35:02.366341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.366311    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:35:02.429348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.429314    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:35:02.479935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.479889    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:35:02.507544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507433    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.507741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507567    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.507741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507670    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.507880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507774    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.507880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507823    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.507880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507873    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.508025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507923    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.508025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.507981    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.508139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.508084    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token\") pod \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\" (UID: \"e81d6aac-e4dc-4f8b-829d-831cf25bfb37\") "
Oct 30 21:35:02.508446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.508362    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.508842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.508809    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.509180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.509157    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.509413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.509357    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.509583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.509426    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.509951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.509926    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.513473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.513440    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.513727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.513668    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.514161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.514138    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "kube-api-access-qqblz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.516028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.516002    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo" pod "e81d6aac-e4dc-4f8b-829d-831cf25bfb37" (UID: "e81d6aac-e4dc-4f8b-829d-831cf25bfb37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:02.559944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.559903    6695 generic.go:356] "Generic (PLEG): container finished" podID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerID="5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301" exitCode=0
Oct 30 21:35:02.560430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.560205    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" event={"ID":"e81d6aac-e4dc-4f8b-829d-831cf25bfb37","Type":"ContainerDied","Data":"5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301"}
Oct 30 21:35:02.560561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.560321    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"
Oct 30 21:35:02.560561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.560505    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv" event={"ID":"e81d6aac-e4dc-4f8b-829d-831cf25bfb37","Type":"ContainerDied","Data":"1cc71b5fb729698474c3a377a5c5f6814aecf37c7daaa5ce70a5ac1deebc38f1"}
Oct 30 21:35:02.560561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.560542    6695 scope.go:122] "RemoveContainer" containerID="5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301"
Oct 30 21:35:02.566361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.565050    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62" event={"ID":"03ac03f9-4225-43cd-bed1-456553318b9f","Type":"ContainerStarted","Data":"a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf"}
Oct 30 21:35:02.566361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.565114    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62" event={"ID":"03ac03f9-4225-43cd-bed1-456553318b9f","Type":"ContainerStarted","Data":"f93eb3e0d56b6ac0bc9c0204f472d8de08cc84ca1905bcd43a074581a537a84a"}
Oct 30 21:35:02.566361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.565928    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:02.571714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.571113    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs" event={"ID":"fccddbf8-0957-46b5-b911-d9f80264887a","Type":"ContainerStarted","Data":"77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087"}
Oct 30 21:35:02.571714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.571171    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs" event={"ID":"fccddbf8-0957-46b5-b911-d9f80264887a","Type":"ContainerStarted","Data":"7bd757526ed489f96a4ba7ed4cdec0ac3d41c3203f0a62571c0e5bd42bf2f1f8"}
Oct 30 21:35:02.571714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.571197    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:02.573486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.573454    6695 generic.go:356] "Generic (PLEG): container finished" podID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerID="7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65" exitCode=0
Oct 30 21:35:02.573675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.573658    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"
Oct 30 21:35:02.573757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.573683    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" event={"ID":"19c587f1-3895-4032-ac4f-623b05bedcb2","Type":"ContainerDied","Data":"7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65"}
Oct 30 21:35:02.573962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.573755    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v" event={"ID":"19c587f1-3895-4032-ac4f-623b05bedcb2","Type":"ContainerDied","Data":"e3796f4a7d30fd13a009de3b87ead08c19c703255041ced0d8943d769ada86b6"}
Oct 30 21:35:02.576021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.575997    6695 generic.go:356] "Generic (PLEG): container finished" podID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerID="b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22" exitCode=0
Oct 30 21:35:02.576223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.576204    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" event={"ID":"b22c3ca7-17a6-4dbd-9121-94137b194f53","Type":"ContainerDied","Data":"b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22"}
Oct 30 21:35:02.576422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.576337    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv" event={"ID":"b22c3ca7-17a6-4dbd-9121-94137b194f53","Type":"ContainerDied","Data":"2e040743c66296e7b1387956f627328987ce7e97ee24df7749efa90af9b7f4dc"}
Oct 30 21:35:02.576561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.576426    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"
Oct 30 21:35:02.578373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.578347    6695 scope.go:122] "RemoveContainer" containerID="5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301"
Oct 30 21:35:02.578681 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:02.578649    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301\": container with ID starting with 5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301 not found: ID does not exist" containerID="5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301"
Oct 30 21:35:02.578767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.578693    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301"} err="failed to get container status \"5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301\": rpc error: code = NotFound desc = could not find container \"5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301\": container with ID starting with 5d6ef7065ddb3103e3306c6a9b541bbe1f9879f1af7257be3200e33a7b78d301 not found: ID does not exist"
Oct 30 21:35:02.578767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.578716    6695 scope.go:122] "RemoveContainer" containerID="7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65"
Oct 30 21:35:02.590505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.590478    6695 scope.go:122] "RemoveContainer" containerID="7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65"
Oct 30 21:35:02.590928 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:02.590899    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65\": container with ID starting with 7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65 not found: ID does not exist" containerID="7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65"
Oct 30 21:35:02.591082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.590939    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65"} err="failed to get container status \"7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65\": rpc error: code = NotFound desc = could not find container \"7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65\": container with ID starting with 7884464bed5276782310118470ac131c1d48446b1f3d8af9f2171453cd359a65 not found: ID does not exist"
Oct 30 21:35:02.591082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.590963    6695 scope.go:122] "RemoveContainer" containerID="b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22"
Oct 30 21:35:02.602235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.602213    6695 scope.go:122] "RemoveContainer" containerID="b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22"
Oct 30 21:35:02.602655 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:02.602592    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22\": container with ID starting with b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22 not found: ID does not exist" containerID="b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22"
Oct 30 21:35:02.602743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.602667    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22"} err="failed to get container status \"b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22\": rpc error: code = NotFound desc = could not find container \"b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22\": container with ID starting with b734c0c78cace3e644527e15c595cb9a317ce2423dbe9da6c081c12c1f7a2c22 not found: ID does not exist"
Oct 30 21:35:02.609806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609772    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.609928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609819    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.609928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609842    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.609928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609871    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.609928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609910    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609944    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.610085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.609974    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610028    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.610085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610065    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.610262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610102    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610107    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.610262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610149    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610195    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.610481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610370    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.610481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610411    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.610481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610449    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610450    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.610811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610482    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610537    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610587    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert\") pod \"19c587f1-3895-4032-ac4f-623b05bedcb2\" (UID: \"19c587f1-3895-4032-ac4f-623b05bedcb2\") "
Oct 30 21:35:02.610811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610746    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket\") pod \"b22c3ca7-17a6-4dbd-9121-94137b194f53\" (UID: \"b22c3ca7-17a6-4dbd-9121-94137b194f53\") "
Oct 30 21:35:02.610811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610759    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.611146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.610809    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.611539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611515    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqblz\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-kube-api-access-qqblz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611543    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611559    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611572    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611586    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611613    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611627    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.611649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611640    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.612012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611653    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22c3ca7-17a6-4dbd-9121-94137b194f53-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.612012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611667    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.612012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611680    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.612012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611692    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.612012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611705    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e81d6aac-e4dc-4f8b-829d-831cf25bfb37-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.612012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.611739    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.612338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.612244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.612338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.612290    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.612638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.612582    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.612719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.612684    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.613640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.613579    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:02.614826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.614770    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:02.614826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.614793    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.614992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.614828    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "kube-api-access-nj5k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.614992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.614958    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.615112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.615087    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml" pod "19c587f1-3895-4032-ac4f-623b05bedcb2" (UID: "19c587f1-3895-4032-ac4f-623b05bedcb2"). InnerVolumeSpecName "kube-api-access-lxrml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.615823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.615801    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:02.616273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.616256    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token" pod "b22c3ca7-17a6-4dbd-9121-94137b194f53" (UID: "b22c3ca7-17a6-4dbd-9121-94137b194f53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.630267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.630208    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs" podStartSLOduration=1.630190016 podStartE2EDuration="1.630190016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:02.627866675 +0000 UTC m=+1485.550324973" watchObservedRunningTime="2026-10-30 21:35:02.630190016 +0000 UTC m=+1485.552648315"
Oct 30 21:35:02.653826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.653781    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"]
Oct 30 21:35:02.658212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.658183    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e38f60-k2csc"]
Oct 30 21:35:02.665915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.665855    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62" podStartSLOduration=1.6658311000000001 podStartE2EDuration="1.6658311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:02.664724497 +0000 UTC m=+1485.587182795" watchObservedRunningTime="2026-10-30 21:35:02.6658311 +0000 UTC m=+1485.588289397"
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.712861    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713016    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713053    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713068    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713081    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713093    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c587f1-3895-4032-ac4f-623b05bedcb2-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713109    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713121    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrml\" (UniqueName: \"kubernetes.io/projected/19c587f1-3895-4032-ac4f-623b05bedcb2-kube-api-access-lxrml\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713133    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22c3ca7-17a6-4dbd-9121-94137b194f53-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713145    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713158    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c587f1-3895-4032-ac4f-623b05bedcb2-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713212    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22c3ca7-17a6-4dbd-9121-94137b194f53-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.713757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.713225    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5k5\" (UniqueName: \"kubernetes.io/projected/b22c3ca7-17a6-4dbd-9121-94137b194f53-kube-api-access-nj5k5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.717832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.717794    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"]
Oct 30 21:35:02.722315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.722277    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-xhqfv"]
Oct 30 21:35:02.924799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.924762    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"]
Oct 30 21:35:02.930798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.930751    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gxjrv"]
Oct 30 21:35:02.991167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:02.991088    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"]
Oct 30 21:35:03.008280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.008235    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gzb2v"]
Oct 30 21:35:03.227338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.227256    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" path="/var/lib/kubelet/pods/19c587f1-3895-4032-ac4f-623b05bedcb2/volumes"
Oct 30 21:35:03.228180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.228159    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad392f41-22a5-4f22-8057-9320f3002821" path="/var/lib/kubelet/pods/ad392f41-22a5-4f22-8057-9320f3002821/volumes"
Oct 30 21:35:03.228883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.228866    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" path="/var/lib/kubelet/pods/b22c3ca7-17a6-4dbd-9121-94137b194f53/volumes"
Oct 30 21:35:03.230011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.229993    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" path="/var/lib/kubelet/pods/e81d6aac-e4dc-4f8b-829d-831cf25bfb37/volumes"
Oct 30 21:35:03.230892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.230867    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"]
Oct 30 21:35:03.231127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:03.231103    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy" containerID="cri-o://981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324" gracePeriod=30
Oct 30 21:35:05.500625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:05.487754    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:35:05.512807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:05.512770    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:35:07.720880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:07.720839    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"]
Oct 30 21:35:07.724562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:07.724531    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1659a-l8bk7"]
Oct 30 21:35:08.521967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.521936    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:35:08.546116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546079    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546201    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546229    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546275    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546340    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546346    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:08.546482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546450    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546487    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546485    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:08.546759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546546    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546630    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy\") pod \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\" (UID: \"8e93ab7d-f44a-4035-b945-63ba14a87d0d\") "
Oct 30 21:35:08.546759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546634    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:08.546983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.546780    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:08.547234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.547210    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.547345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.547242    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.547345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.547259    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.547345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.547267    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.548071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.548047    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:08.551647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.549729    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:08.551647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.549764    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:08.551647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.549762    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:08.551647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.549784    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl" pod "8e93ab7d-f44a-4035-b945-63ba14a87d0d" (UID: "8e93ab7d-f44a-4035-b945-63ba14a87d0d"). InnerVolumeSpecName "kube-api-access-clfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:08.648901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.648860    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.648901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.648891    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.648901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.648904    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.649144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.648916    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfwl\" (UniqueName: \"kubernetes.io/projected/8e93ab7d-f44a-4035-b945-63ba14a87d0d-kube-api-access-clfwl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.649144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.648929    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e93ab7d-f44a-4035-b945-63ba14a87d0d-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:08.737667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.737626    6695 generic.go:356] "Generic (PLEG): container finished" podID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerID="981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324" exitCode=0
Oct 30 21:35:08.738148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.737687    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"
Oct 30 21:35:08.738148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.737716    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" event={"ID":"8e93ab7d-f44a-4035-b945-63ba14a87d0d","Type":"ContainerDied","Data":"981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324"}
Oct 30 21:35:08.738148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.737767    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn" event={"ID":"8e93ab7d-f44a-4035-b945-63ba14a87d0d","Type":"ContainerDied","Data":"385f7a4c472d74ca12abb3bdf7d1c9c17a25f8e53dbc5e0a66423fdc6fa2521d"}
Oct 30 21:35:08.738148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.737787    6695 scope.go:122] "RemoveContainer" containerID="981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324"
Oct 30 21:35:08.752402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.750756    6695 scope.go:122] "RemoveContainer" containerID="0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"
Oct 30 21:35:08.763821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.763802    6695 scope.go:122] "RemoveContainer" containerID="981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324"
Oct 30 21:35:08.764131 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:08.764114    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324\": container with ID starting with 981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324 not found: ID does not exist" containerID="981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324"
Oct 30 21:35:08.764178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.764139    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324"} err="failed to get container status \"981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324\": rpc error: code = NotFound desc = could not find container \"981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324\": container with ID starting with 981ee9e7bc0b27be0fef7977cd24c51d868d205e5cc89dead917f0fe34afd324 not found: ID does not exist"
Oct 30 21:35:08.764178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.764156    6695 scope.go:122] "RemoveContainer" containerID="0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"
Oct 30 21:35:08.764381 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:08.764364    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793\": container with ID starting with 0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793 not found: ID does not exist" containerID="0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"
Oct 30 21:35:08.764425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.764387    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793"} err="failed to get container status \"0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793\": rpc error: code = NotFound desc = could not find container \"0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793\": container with ID starting with 0594e85f44c5c30c732c0a4d2ca0c9357b279c0ec32998e899b379db087d5793 not found: ID does not exist"
Oct 30 21:35:08.809036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.808998    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"]
Oct 30 21:35:08.809266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:08.809056    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7222e2-openshift-default-f44fd5dd9-sflbn"]
Oct 30 21:35:09.214086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:09.214053    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7697dfb3-a743-45ea-963b-199c9eebb4db" path="/var/lib/kubelet/pods/7697dfb3-a743-45ea-963b-199c9eebb4db/volumes"
Oct 30 21:35:09.214591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:09.214576    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" path="/var/lib/kubelet/pods/8e93ab7d-f44a-4035-b945-63ba14a87d0d/volumes"
Oct 30 21:35:10.689424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.689382    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"]
Oct 30 21:35:10.690195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690172    6695 cpu_manager.go:414] "containerMap: removing container" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy"
Oct 30 21:35:10.690302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690196    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy"
Oct 30 21:35:10.690302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690221    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy"
Oct 30 21:35:10.690302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690228    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy"
Oct 30 21:35:10.690302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690244    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy"
Oct 30 21:35:10.690302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690252    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy"
Oct 30 21:35:10.690302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690301    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy"
Oct 30 21:35:10.690653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690308    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy"
Oct 30 21:35:10.690653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690440    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy"
Oct 30 21:35:10.690653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690456    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22c3ca7-17a6-4dbd-9121-94137b194f53" containerName="istio-proxy"
Oct 30 21:35:10.690653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690470    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy"
Oct 30 21:35:10.690653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690487    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81d6aac-e4dc-4f8b-829d-831cf25bfb37" containerName="istio-proxy"
Oct 30 21:35:10.690653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.690501    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c587f1-3895-4032-ac4f-623b05bedcb2" containerName="istio-proxy"
Oct 30 21:35:10.695515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.695492    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.715329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.715300    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"]
Oct 30 21:35:10.730642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.730230    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:35:10.730642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.730251    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:35:10.748323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.748289    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:35:10.750033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.750009    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:35:10.787630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.787548    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.787630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.787588    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2v84\" (UniqueName: \"kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.787911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.787767    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.881387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.881344    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"]
Oct 30 21:35:10.881598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.881570    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy" containerID="cri-o://ba45d4e95e06a74ae0b078314d8034881c342c355f13af664e58f1a27aee1b9c" gracePeriod=30
Oct 30 21:35:10.888718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.888684    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.888943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.888925    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.889016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.888963    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2v84\" (UniqueName: \"kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.890260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.890233    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.892132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.892106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:10.913429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:10.913386    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2v84\" (UniqueName: \"kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84\") pod \"maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:11.010280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.010180    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:11.317094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.316978    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"]
Oct 30 21:35:11.321455 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:11.321422    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b937cc_0b95_4612_859f_7cb9bcc4e5b1.slice/crio-92f1be3ccc4be35a088b453aa0ea482e6652be3768abc17bd5f46334445d953b WatchSource:0}: Error finding container 92f1be3ccc4be35a088b453aa0ea482e6652be3768abc17bd5f46334445d953b: Status 404 returned error can't find the container with id 92f1be3ccc4be35a088b453aa0ea482e6652be3768abc17bd5f46334445d953b
Oct 30 21:35:11.631843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.631816    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"]
Oct 30 21:35:11.632697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.632674    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy"
Oct 30 21:35:11.632697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.632695    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e93ab7d-f44a-4035-b945-63ba14a87d0d" containerName="istio-proxy"
Oct 30 21:35:11.636228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.636209    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.651392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.651369    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3a5c92-dockercfg-llg7k\""
Oct 30 21:35:11.654696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.654678    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3a5c92\""
Oct 30 21:35:11.661380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.661351    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"]
Oct 30 21:35:11.713588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.713548    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.738626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.736682    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756g9\" (UniqueName: \"kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9\") pod \"payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.886261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.886157    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"]
Oct 30 21:35:11.895717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.895681    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.896161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.896124    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756g9\" (UniqueName: \"kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9\") pod \"payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.898020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.897986    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.906113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.905962    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"]
Oct 30 21:35:11.906113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.906080    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:11.911631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.910841    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7" event={"ID":"75b937cc-0b95-4612-859f-7cb9bcc4e5b1","Type":"ContainerStarted","Data":"61ef3e18d4a757ab9bcfca29d0c43ff7aa768292bb561b1d24592492b463673d"}
Oct 30 21:35:11.911631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.910894    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7" event={"ID":"75b937cc-0b95-4612-859f-7cb9bcc4e5b1","Type":"ContainerStarted","Data":"92f1be3ccc4be35a088b453aa0ea482e6652be3768abc17bd5f46334445d953b"}
Oct 30 21:35:11.919824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.919672    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756g9\" (UniqueName: \"kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9\") pod \"payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.953076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.953039    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:11.993598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:11.993521    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7" podStartSLOduration=1.993491135 podStartE2EDuration="1.993491135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:11.991050493 +0000 UTC m=+1494.913508791" watchObservedRunningTime="2026-10-30 21:35:11.993491135 +0000 UTC m=+1494.915949435"
Oct 30 21:35:12.004581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.003419    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9r2\" (UniqueName: \"kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2\") pod \"payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.004581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.003545    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.105425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.105380    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9r2\" (UniqueName: \"kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2\") pod \"payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.105639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.105467    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.106509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.106478    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.133101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.131719    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9r2\" (UniqueName: \"kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2\") pod \"payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.242105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.242016    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.396664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.395032    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"]
Oct 30 21:35:12.408350 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:12.407071    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc4f993_e60a_4ecd_8bca_156ef48091a8.slice/crio-3f48dcdc598e62d48f623b7082ff22ea1b38cdd933ef46f39e4fa88641b5a223 WatchSource:0}: Error finding container 3f48dcdc598e62d48f623b7082ff22ea1b38cdd933ef46f39e4fa88641b5a223: Status 404 returned error can't find the container with id 3f48dcdc598e62d48f623b7082ff22ea1b38cdd933ef46f39e4fa88641b5a223
Oct 30 21:35:12.680997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.680962    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"]
Oct 30 21:35:12.684652 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:12.684626    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0937ce8_ff47_49dc_b7ea_057d1ba37168.slice/crio-38d742a49fb0ad31aa675b0328d1b133e5a58a0dda0cb21e8e97d5837c8f82fd WatchSource:0}: Error finding container 38d742a49fb0ad31aa675b0328d1b133e5a58a0dda0cb21e8e97d5837c8f82fd: Status 404 returned error can't find the container with id 38d742a49fb0ad31aa675b0328d1b133e5a58a0dda0cb21e8e97d5837c8f82fd
Oct 30 21:35:12.920651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.920597    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n" event={"ID":"0cc4f993-e60a-4ecd-8bca-156ef48091a8","Type":"ContainerStarted","Data":"5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4"}
Oct 30 21:35:12.921155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.920723    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n" event={"ID":"0cc4f993-e60a-4ecd-8bca-156ef48091a8","Type":"ContainerStarted","Data":"3f48dcdc598e62d48f623b7082ff22ea1b38cdd933ef46f39e4fa88641b5a223"}
Oct 30 21:35:12.921155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.920752    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:12.922512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.922477    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68" event={"ID":"f0937ce8-ff47-49dc-b7ea-057d1ba37168","Type":"ContainerStarted","Data":"781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb"}
Oct 30 21:35:12.922674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.922541    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68" event={"ID":"f0937ce8-ff47-49dc-b7ea-057d1ba37168","Type":"ContainerStarted","Data":"38d742a49fb0ad31aa675b0328d1b133e5a58a0dda0cb21e8e97d5837c8f82fd"}
Oct 30 21:35:12.922674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.922571    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:12.924286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.924255    6695 generic.go:356] "Generic (PLEG): container finished" podID="75b937cc-0b95-4612-859f-7cb9bcc4e5b1" containerID="61ef3e18d4a757ab9bcfca29d0c43ff7aa768292bb561b1d24592492b463673d" exitCode=0
Oct 30 21:35:12.924391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.924338    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7" event={"ID":"75b937cc-0b95-4612-859f-7cb9bcc4e5b1","Type":"ContainerDied","Data":"61ef3e18d4a757ab9bcfca29d0c43ff7aa768292bb561b1d24592492b463673d"}
Oct 30 21:35:12.983690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:12.983585    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n" podStartSLOduration=1.983573762 podStartE2EDuration="1.983573762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:12.982754087 +0000 UTC m=+1495.905212385" watchObservedRunningTime="2026-10-30 21:35:12.983573762 +0000 UTC m=+1495.906032060"
Oct 30 21:35:13.117955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:13.117897    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68" podStartSLOduration=2.117883552 podStartE2EDuration="2.117883552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:13.114778455 +0000 UTC m=+1496.037236754" watchObservedRunningTime="2026-10-30 21:35:13.117883552 +0000 UTC m=+1496.040341849"
Oct 30 21:35:13.589850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:13.589815    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:35:14.092545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.092519    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:14.140997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.140964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca\") pod \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") "
Oct 30 21:35:14.141187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.141081    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca\") pod \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") "
Oct 30 21:35:14.141247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.141202    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84\" (UniqueName: \"kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84\") pod \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\" (UID: \"75b937cc-0b95-4612-859f-7cb9bcc4e5b1\") "
Oct 30 21:35:14.141430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.141406    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca" pod "75b937cc-0b95-4612-859f-7cb9bcc4e5b1" (UID: "75b937cc-0b95-4612-859f-7cb9bcc4e5b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:14.141685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.141668    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:14.142997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.142971    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca" pod "75b937cc-0b95-4612-859f-7cb9bcc4e5b1" (UID: "75b937cc-0b95-4612-859f-7cb9bcc4e5b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:14.143185 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.143168    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84" pod "75b937cc-0b95-4612-859f-7cb9bcc4e5b1" (UID: "75b937cc-0b95-4612-859f-7cb9bcc4e5b1"). InnerVolumeSpecName "kube-api-access-x2v84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:14.222995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.222912    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"]
Oct 30 21:35:14.223855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.223830    6695 cpu_manager.go:414] "containerMap: removing container" podUID="75b937cc-0b95-4612-859f-7cb9bcc4e5b1" containerName="revoke-keys"
Oct 30 21:35:14.223940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.223856    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b937cc-0b95-4612-859f-7cb9bcc4e5b1" containerName="revoke-keys"
Oct 30 21:35:14.224109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.224094    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b937cc-0b95-4612-859f-7cb9bcc4e5b1" containerName="revoke-keys"
Oct 30 21:35:14.228122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.228089    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242324    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242473    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242571    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242650    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-13658f-openshift-default-dockercfg-jng8c\""
Oct 30 21:35:14.242666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242656    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242728    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv87m\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242746    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242827    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242866    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.242972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.242959    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.243060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.243049    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2v84\" (UniqueName: \"kubernetes.io/projected/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-kube-api-access-x2v84\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:14.243098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.243068    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b937cc-0b95-4612-859f-7cb9bcc4e5b1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:14.257351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.257321    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"]
Oct 30 21:35:14.344576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv87m\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.344576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344579    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.344830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344639    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.344830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344678    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.344830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344761    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344858    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.344974    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345019    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345039    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345069    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345072    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345360    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345410    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.345513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.345492    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.347035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.347012    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.347358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.347337    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.365237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.365212    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv87m\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.365350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.365243    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token\") pod \"e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") " pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.546154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.544890    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:14.597934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.597456    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:35:14.926010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.925969    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"]
Oct 30 21:35:14.998752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.998684    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:14.999164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:14.999123    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:15.008142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:15.008105    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:15.020929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:15.018851    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7" event={"ID":"75b937cc-0b95-4612-859f-7cb9bcc4e5b1","Type":"ContainerDied","Data":"92f1be3ccc4be35a088b453aa0ea482e6652be3768abc17bd5f46334445d953b"}
Oct 30 21:35:15.020929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:15.018910    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f1be3ccc4be35a088b453aa0ea482e6652be3768abc17bd5f46334445d953b"
Oct 30 21:35:15.020929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:15.018992    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"
Oct 30 21:35:15.026055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:15.026010    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" event={"ID":"b49aa851-f301-40f5-bc59-cc2fb9b80ccf","Type":"ContainerStarted","Data":"cfeea236d932cbd5b2a4f5195d0a00f122b4610526e66d3b022692e0e29e45b3"}
Oct 30 21:35:16.036522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.036488    6695 generic.go:356] "Generic (PLEG): container finished" podID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerID="ba45d4e95e06a74ae0b078314d8034881c342c355f13af664e58f1a27aee1b9c" exitCode=0
Oct 30 21:35:16.037051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.036567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" event={"ID":"244dad53-4bc6-4800-a237-25ed5be27b5d","Type":"ContainerDied","Data":"ba45d4e95e06a74ae0b078314d8034881c342c355f13af664e58f1a27aee1b9c"}
Oct 30 21:35:16.037051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.036643    6695 scope.go:122] "RemoveContainer" containerID="6d8bec46dcdf1c45bfe92b7566273f3f21fb15e2053c6879d67f2d45f675d6ab"
Oct 30 21:35:16.038812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.038591    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" event={"ID":"b49aa851-f301-40f5-bc59-cc2fb9b80ccf","Type":"ContainerStarted","Data":"acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"}
Oct 30 21:35:16.116066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.116007    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podStartSLOduration=2.115989048 podStartE2EDuration="2.115989048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:16.102066015 +0000 UTC m=+1499.024524350" watchObservedRunningTime="2026-10-30 21:35:16.115989048 +0000 UTC m=+1499.038447346"
Oct 30 21:35:16.171679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.171653    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:35:16.294459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294421    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.294675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294487    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.294675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294555    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.294675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294661    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.294853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294737    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.294853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294777    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.294853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294845    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.295003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294870    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:16.295003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294936    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.295003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.294941    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:16.295203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295030    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token\") pod \"244dad53-4bc6-4800-a237-25ed5be27b5d\" (UID: \"244dad53-4bc6-4800-a237-25ed5be27b5d\") "
Oct 30 21:35:16.295203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295075    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:16.295412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295389    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:16.295837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295704    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.295837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295731    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244dad53-4bc6-4800-a237-25ed5be27b5d-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.295837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295760    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.295837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295786    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.295837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.295815    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:16.297935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.297888    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:16.298152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.298118    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:16.298373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.298349    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:16.298822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.298790    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj" pod "244dad53-4bc6-4800-a237-25ed5be27b5d" (UID: "244dad53-4bc6-4800-a237-25ed5be27b5d"). InnerVolumeSpecName "kube-api-access-slfmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:16.401226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.401052    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfmj\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-kube-api-access-slfmj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.401226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.401093    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.401226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.401107    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244dad53-4bc6-4800-a237-25ed5be27b5d-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.401226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.401122    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.401226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.401136    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244dad53-4bc6-4800-a237-25ed5be27b5d-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:16.557632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.555493    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:16.584624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.575938    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:16.645319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.645288    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"]
Oct 30 21:35:16.645526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:16.645501    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy" containerID="cri-o://fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0" gracePeriod=30
Oct 30 21:35:17.048039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.048008    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"
Oct 30 21:35:17.048519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.048040    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl" event={"ID":"244dad53-4bc6-4800-a237-25ed5be27b5d","Type":"ContainerDied","Data":"8213d96a4307eccb49bd11d6349796836f6d5b71009e0cefe16488d947a48ef1"}
Oct 30 21:35:17.048519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.048106    6695 scope.go:122] "RemoveContainer" containerID="ba45d4e95e06a74ae0b078314d8034881c342c355f13af664e58f1a27aee1b9c"
Oct 30 21:35:17.048662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.048566    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:17.049763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.049726    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:35:17.108497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.108464    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"]
Oct 30 21:35:17.118525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.118493    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ztwxl"]
Oct 30 21:35:17.218978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.218948    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" path="/var/lib/kubelet/pods/244dad53-4bc6-4800-a237-25ed5be27b5d/volumes"
Oct 30 21:35:17.478172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.478092    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:35:17.506964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:17.506931    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:35:18.702153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:18.702115    6695 scope.go:122] "RemoveContainer" containerID="5455fd04cedf5d7de5abb71dac8d45441efe73d9fdad1fb29aac9169ed169996"
Oct 30 21:35:18.713056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:18.713031    6695 scope.go:122] "RemoveContainer" containerID="e7773d1e6e98c0dd5c868c289410680075a20eaf48b8423a05be20daba15f650"
Oct 30 21:35:18.723469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:18.723444    6695 scope.go:122] "RemoveContainer" containerID="246370eb3de0b93e51ac9fae87d9220fbdd838edf0ba624c8f3acffd6e45e63a"
Oct 30 21:35:18.734440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:18.734415    6695 scope.go:122] "RemoveContainer" containerID="0bfe1e4553c0f08c36f37b2f6eb9b6d28b92f2a151d6553e4cce8cc19ef3519e"
Oct 30 21:35:19.194823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.193939    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"]
Oct 30 21:35:19.195251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.195226    6695 cpu_manager.go:414] "containerMap: removing container" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy"
Oct 30 21:35:19.195251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.195251    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy"
Oct 30 21:35:19.195435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.195275    6695 cpu_manager.go:414] "containerMap: removing container" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy"
Oct 30 21:35:19.195435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.195282    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy"
Oct 30 21:35:19.195531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.195439    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy"
Oct 30 21:35:19.195531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.195470    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="244dad53-4bc6-4800-a237-25ed5be27b5d" containerName="istio-proxy"
Oct 30 21:35:19.202795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.202761    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.239967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.239941    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-26a13f-openshift-default-dockercfg-lhwhv\""
Oct 30 21:35:19.251866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.251838    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.251875    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.251894    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.252046    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.252077    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.252273    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.252299    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.252378    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlqq\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.252518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.252500    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359104    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359170    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359193    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359274    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359300    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359423    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359451    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlqq\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.359577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.359562    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.363835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.360961    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.363835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.361400    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.363835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.361897    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.363835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.362383    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.363835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.362676    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.363835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.363757    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.394626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.383915    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.424641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.415949    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.435301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.435260    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlqq\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq\") pod \"e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") " pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.451533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.451438    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"]
Oct 30 21:35:19.528888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.528849    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:19.878577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.878546    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"]
Oct 30 21:35:19.882990 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:19.882952    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ac6495_1f88_4207_ad29_92fd7e2aba41.slice/crio-a10c1051c1104927b59cedec3558d64fde7f4644f6e3b1a323ba6aaf7aac8ad2 WatchSource:0}: Error finding container a10c1051c1104927b59cedec3558d64fde7f4644f6e3b1a323ba6aaf7aac8ad2: Status 404 returned error can't find the container with id a10c1051c1104927b59cedec3558d64fde7f4644f6e3b1a323ba6aaf7aac8ad2
Oct 30 21:35:19.885644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.885579    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:19.885725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.885667    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:19.885725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:19.885705    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:20.116079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:20.116036    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" event={"ID":"37ac6495-1f88-4207-ad29-92fd7e2aba41","Type":"ContainerStarted","Data":"546ce07ba3010e65fd9fb89b79cf3cc4f930ae8801c4f0c7a61dac5535f84f4e"}
Oct 30 21:35:20.116286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:20.116109    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" event={"ID":"37ac6495-1f88-4207-ad29-92fd7e2aba41","Type":"ContainerStarted","Data":"a10c1051c1104927b59cedec3558d64fde7f4644f6e3b1a323ba6aaf7aac8ad2"}
Oct 30 21:35:20.176637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:20.176490    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podStartSLOduration=1.1764693290000001 podStartE2EDuration="1.176469329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:20.175407859 +0000 UTC m=+1503.097866157" watchObservedRunningTime="2026-10-30 21:35:20.176469329 +0000 UTC m=+1503.098927628"
Oct 30 21:35:20.529505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:20.529470    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:20.535095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:20.535068    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:21.123529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:21.123492    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:21.124950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:21.124927    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:35:21.972626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:21.972588    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:35:22.025205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025171    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025222    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025305    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025360    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025476    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025515    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025552    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025658    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket\") pod \"22ba006c-a1ed-4618-81b6-91144a0c50a3\" (UID: \"22ba006c-a1ed-4618-81b6-91144a0c50a3\") "
Oct 30 21:35:22.025737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025694    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:22.025979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025797    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:22.025979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.025813    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:22.026088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026032    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:22.026196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026177    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:22.026319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026303    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ba006c-a1ed-4618-81b6-91144a0c50a3-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.026401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026324    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.026401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026338    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.026401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026351    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.026401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.026365    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.027776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.027747    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:22.027875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.027852    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "kube-api-access-d8pzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:22.027994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.027971    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:22.028065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.028034    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token" pod "22ba006c-a1ed-4618-81b6-91144a0c50a3" (UID: "22ba006c-a1ed-4618-81b6-91144a0c50a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:22.127365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.127333    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8pzd\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-kube-api-access-d8pzd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.127365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.127361    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.127365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.127371    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.127885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.127381    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ba006c-a1ed-4618-81b6-91144a0c50a3-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:22.132005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.131974    6695 generic.go:356] "Generic (PLEG): container finished" podID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerID="fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0" exitCode=0
Oct 30 21:35:22.132152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.132015    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" event={"ID":"22ba006c-a1ed-4618-81b6-91144a0c50a3","Type":"ContainerDied","Data":"fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0"}
Oct 30 21:35:22.132152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.132066    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"
Oct 30 21:35:22.132152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.132085    6695 scope.go:122] "RemoveContainer" containerID="fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0"
Oct 30 21:35:22.132322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.132072    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh" event={"ID":"22ba006c-a1ed-4618-81b6-91144a0c50a3","Type":"ContainerDied","Data":"ba73a6df53122b1e2e767df23f5dbeb02e945f56731ce9438f3ec27b44a6a70c"}
Oct 30 21:35:22.143191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.143172    6695 scope.go:122] "RemoveContainer" containerID="953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"
Oct 30 21:35:22.153195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.153175    6695 scope.go:122] "RemoveContainer" containerID="fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0"
Oct 30 21:35:22.153496 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:22.153479    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0\": container with ID starting with fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0 not found: ID does not exist" containerID="fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0"
Oct 30 21:35:22.153573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.153505    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0"} err="failed to get container status \"fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0\": rpc error: code = NotFound desc = could not find container \"fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0\": container with ID starting with fa9d89897bc81aa4610817f7a6d585d5f07fc047cdd7d5d8698a8bc2541ddfd0 not found: ID does not exist"
Oct 30 21:35:22.153573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.153521    6695 scope.go:122] "RemoveContainer" containerID="953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"
Oct 30 21:35:22.153811 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:22.153789    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442\": container with ID starting with 953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442 not found: ID does not exist" containerID="953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"
Oct 30 21:35:22.153855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.153818    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442"} err="failed to get container status \"953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442\": rpc error: code = NotFound desc = could not find container \"953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442\": container with ID starting with 953f03b1c36a12a793d0e7d74892842745fd30f3faa9245f98a88f3f0d48d442 not found: ID does not exist"
Oct 30 21:35:22.204286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.204189    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"]
Oct 30 21:35:22.207013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.206989    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b1659a-openshift-default-594764768c-v58qh"]
Oct 30 21:35:22.535336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:22.535305    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs_fccddbf8-0957-46b5-b911-d9f80264887a/payload-processing/0.log"
Oct 30 21:35:23.053374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:23.053335    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"]
Oct 30 21:35:23.065220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:23.065187    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6cd92d4-962mb"]
Oct 30 21:35:23.214281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:23.214248    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" path="/var/lib/kubelet/pods/22ba006c-a1ed-4618-81b6-91144a0c50a3/volumes"
Oct 30 21:35:23.214808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:23.214790    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9d02c2-695a-418f-8c9b-79412b38c072" path="/var/lib/kubelet/pods/8e9d02c2-695a-418f-8c9b-79412b38c072/volumes"
Oct 30 21:35:23.935426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:23.935393    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:35:23.935654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:23.935520    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:35:24.936241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.936207    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"]
Oct 30 21:35:24.936974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.936945    6695 cpu_manager.go:414] "containerMap: removing container" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy"
Oct 30 21:35:24.936974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.936963    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy"
Oct 30 21:35:24.937136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.937003    6695 cpu_manager.go:414] "containerMap: removing container" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy"
Oct 30 21:35:24.937136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.937009    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy"
Oct 30 21:35:24.937136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.937108    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy"
Oct 30 21:35:24.937136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.937123    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ba006c-a1ed-4618-81b6-91144a0c50a3" containerName="istio-proxy"
Oct 30 21:35:24.942689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.942666    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:24.968575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.968542    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"]
Oct 30 21:35:24.972937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.972914    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:24.975555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.975508    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-26a13f-dockercfg-r9b5h\""
Oct 30 21:35:24.975555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.975521    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-26a13f\""
Oct 30 21:35:24.988201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:24.988172    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"]
Oct 30 21:35:25.002847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.002815    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"]
Oct 30 21:35:25.064923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.064891    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0a1ee3\"/\"kube-root-ca.crt\""
Oct 30 21:35:25.064923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.064891    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0a1ee3\"/\"openshift-service-ca.crt\""
Oct 30 21:35:25.065211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.065165    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0a1ee3\"/\"default-dockercfg-tbfrt\""
Oct 30 21:35:25.065282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.065230    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0a1ee3\"/\"ipp-route-0a1ee3-kserve-self-signed-certs\""
Oct 30 21:35:25.089858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.089821    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.090217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090184    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.090350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090265    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.090350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090317    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.090350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090341    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.090504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090433    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gc4r\" (UniqueName: \"kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r\") pod \"payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.090593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090572    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54bb\" (UniqueName: \"kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.090695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.090675    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.191528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191448    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.191528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191494    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.191789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191553    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.191789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191576    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.191789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191664    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gc4r\" (UniqueName: \"kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r\") pod \"payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.191789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191753    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54bb\" (UniqueName: \"kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.191789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191786    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.192060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191841    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.192060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.191961    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.192060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.192001    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.192060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.192017    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.192642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.192599    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.193906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.193887    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.194285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.194268    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.219737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.219708    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gc4r\" (UniqueName: \"kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r\") pod \"payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.219931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.219909    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54bb\" (UniqueName: \"kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb\") pod \"ipp-route-0a1ee3-kserve-d44874c5c-lkhg9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") " pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.284626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.281712    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:25.295627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.290572    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:25.748540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.748460    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"]
Oct 30 21:35:25.752414 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:25.752387    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a08fad2_c55e_4560_b7b5_0eddf63903a5.slice/crio-081eccebe5b269dbefb0241d6cd352252056932fc4a0d03ff2b7c35997054f10 WatchSource:0}: Error finding container 081eccebe5b269dbefb0241d6cd352252056932fc4a0d03ff2b7c35997054f10: Status 404 returned error can't find the container with id 081eccebe5b269dbefb0241d6cd352252056932fc4a0d03ff2b7c35997054f10
Oct 30 21:35:25.755388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:25.755363    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"]
Oct 30 21:35:25.759658 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:25.759629    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6ee646f_f360_4105_a775_a82d607efbf9.slice/crio-7baa8bf4697a8db67dd6d1cbb0708560ba4aed2856232be3ba23fbba8e6c93ab WatchSource:0}: Error finding container 7baa8bf4697a8db67dd6d1cbb0708560ba4aed2856232be3ba23fbba8e6c93ab: Status 404 returned error can't find the container with id 7baa8bf4697a8db67dd6d1cbb0708560ba4aed2856232be3ba23fbba8e6c93ab
Oct 30 21:35:26.163788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.163749    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9" event={"ID":"a6ee646f-f360-4105-a775-a82d607efbf9","Type":"ContainerStarted","Data":"f43129d133308c7ec30d7b105ac74e6fdffa258725753c70cf957a0ac7d104d2"}
Oct 30 21:35:26.164223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.163796    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9" event={"ID":"a6ee646f-f360-4105-a775-a82d607efbf9","Type":"ContainerStarted","Data":"7baa8bf4697a8db67dd6d1cbb0708560ba4aed2856232be3ba23fbba8e6c93ab"}
Oct 30 21:35:26.165568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.165544    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph" event={"ID":"4a08fad2-c55e-4560-b7b5-0eddf63903a5","Type":"ContainerStarted","Data":"d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea"}
Oct 30 21:35:26.165702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.165596    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:26.165702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.165640    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph" event={"ID":"4a08fad2-c55e-4560-b7b5-0eddf63903a5","Type":"ContainerStarted","Data":"081eccebe5b269dbefb0241d6cd352252056932fc4a0d03ff2b7c35997054f10"}
Oct 30 21:35:26.230134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.230076    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9" podStartSLOduration=2.230062093 podStartE2EDuration="2.230062093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.22812739 +0000 UTC m=+1509.150585688" watchObservedRunningTime="2026-10-30 21:35:26.230062093 +0000 UTC m=+1509.152520390"
Oct 30 21:35:26.282786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.282706    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph" podStartSLOduration=2.282692179 podStartE2EDuration="2.282692179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.280280496 +0000 UTC m=+1509.202738793" watchObservedRunningTime="2026-10-30 21:35:26.282692179 +0000 UTC m=+1509.205150480"
Oct 30 21:35:26.291504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:26.291475    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:31.074263 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:31.074219    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:35:35.290955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:35.290917    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:35.305095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:35.305066    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:35.316217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:35.316190    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:35:37.175369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:37.175336    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:35:40.985785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:40.985748    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"]
Oct 30 21:35:40.986285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:40.985952    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy" containerID="cri-o://158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6" gracePeriod=30
Oct 30 21:35:43.509121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:43.509079    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"]
Oct 30 21:35:43.509593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:43.509376    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" podUID="5374b820-82f7-47ab-a547-ea7a2563331c" containerName="payload-processing" containerID="cri-o://35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003" gracePeriod=30
Oct 30 21:35:43.550619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:43.550568    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"]
Oct 30 21:35:43.551246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:43.551215    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" podUID="78d5fce9-d075-4ece-849a-8212323a51a2" containerName="payload-pre-processing" containerID="cri-o://474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2" gracePeriod=30
Oct 30 21:35:43.656318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:43.656273    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" podUID="78d5fce9-d075-4ece-849a-8212323a51a2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.162:9004: connect: connection refused"
Oct 30 21:35:43.656491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:43.656288    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" podUID="5374b820-82f7-47ab-a547-ea7a2563331c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.161:9004: connect: connection refused"
Oct 30 21:35:44.067039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.067010    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:35:44.084657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.084628    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:35:44.164441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.164398    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume\") pod \"5374b820-82f7-47ab-a547-ea7a2563331c\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") "
Oct 30 21:35:44.164737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.164716    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx\") pod \"5374b820-82f7-47ab-a547-ea7a2563331c\" (UID: \"5374b820-82f7-47ab-a547-ea7a2563331c\") "
Oct 30 21:35:44.164888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.164797    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt\" (UniqueName: \"kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt\") pod \"78d5fce9-d075-4ece-849a-8212323a51a2\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") "
Oct 30 21:35:44.164888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.164837    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume\") pod \"78d5fce9-d075-4ece-849a-8212323a51a2\" (UID: \"78d5fce9-d075-4ece-849a-8212323a51a2\") "
Oct 30 21:35:44.165712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.165666    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume" pod "5374b820-82f7-47ab-a547-ea7a2563331c" (UID: "5374b820-82f7-47ab-a547-ea7a2563331c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:44.165712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.165704    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume" pod "78d5fce9-d075-4ece-849a-8212323a51a2" (UID: "78d5fce9-d075-4ece-849a-8212323a51a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:44.172655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.169278    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt" pod "78d5fce9-d075-4ece-849a-8212323a51a2" (UID: "78d5fce9-d075-4ece-849a-8212323a51a2"). InnerVolumeSpecName "kube-api-access-gdjxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:44.172655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.169848    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx" pod "5374b820-82f7-47ab-a547-ea7a2563331c" (UID: "5374b820-82f7-47ab-a547-ea7a2563331c"). InnerVolumeSpecName "kube-api-access-8mfrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:44.267135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.267099    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfrx\" (UniqueName: \"kubernetes.io/projected/5374b820-82f7-47ab-a547-ea7a2563331c-kube-api-access-8mfrx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:44.267135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.267130    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjxt\" (UniqueName: \"kubernetes.io/projected/78d5fce9-d075-4ece-849a-8212323a51a2-kube-api-access-gdjxt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:44.267135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.267144    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78d5fce9-d075-4ece-849a-8212323a51a2-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:44.267398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.267156    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5374b820-82f7-47ab-a547-ea7a2563331c-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:44.328146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.328071    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"]
Oct 30 21:35:44.332208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.332178    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3afc5e-bbfx7"]
Oct 30 21:35:44.368420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.368379    6695 generic.go:356] "Generic (PLEG): container finished" podID="78d5fce9-d075-4ece-849a-8212323a51a2" containerID="474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2" exitCode=0
Oct 30 21:35:44.368693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.368450    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"
Oct 30 21:35:44.368693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.368575    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" event={"ID":"78d5fce9-d075-4ece-849a-8212323a51a2","Type":"ContainerDied","Data":"474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2"}
Oct 30 21:35:44.368693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.368648    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp" event={"ID":"78d5fce9-d075-4ece-849a-8212323a51a2","Type":"ContainerDied","Data":"9b12cbce89959876057e1ff64050541587124995eca5e49ddf0ea0b934c79f6a"}
Oct 30 21:35:44.368693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.368678    6695 scope.go:122] "RemoveContainer" containerID="474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2"
Oct 30 21:35:44.370534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.370514    6695 generic.go:356] "Generic (PLEG): container finished" podID="5374b820-82f7-47ab-a547-ea7a2563331c" containerID="35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003" exitCode=0
Oct 30 21:35:44.370647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.370560    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"
Oct 30 21:35:44.370647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.370556    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" event={"ID":"5374b820-82f7-47ab-a547-ea7a2563331c","Type":"ContainerDied","Data":"35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003"}
Oct 30 21:35:44.370771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.370674    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv" event={"ID":"5374b820-82f7-47ab-a547-ea7a2563331c","Type":"ContainerDied","Data":"8e90e311db8fc002141a285c7c57388a257fce889a09232ad561bd7cac9ffb24"}
Oct 30 21:35:44.381783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.381763    6695 scope.go:122] "RemoveContainer" containerID="474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2"
Oct 30 21:35:44.382093 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:44.382071    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2\": container with ID starting with 474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2 not found: ID does not exist" containerID="474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2"
Oct 30 21:35:44.382192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.382101    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2"} err="failed to get container status \"474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2\": rpc error: code = NotFound desc = could not find container \"474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2\": container with ID starting with 474a79258f2ba0e78caf040e68a40545d0018cbd71c8e1755d7c107a4a5113f2 not found: ID does not exist"
Oct 30 21:35:44.382192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.382124    6695 scope.go:122] "RemoveContainer" containerID="35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003"
Oct 30 21:35:44.396546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.396494    6695 scope.go:122] "RemoveContainer" containerID="35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003"
Oct 30 21:35:44.396922 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:44.396899    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003\": container with ID starting with 35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003 not found: ID does not exist" containerID="35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003"
Oct 30 21:35:44.397029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.396931    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003"} err="failed to get container status \"35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003\": rpc error: code = NotFound desc = could not find container \"35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003\": container with ID starting with 35a7a18d1c0e6cdf95e2667a2f4abc0a2014e1bff6da7d020a41a2614c55a003 not found: ID does not exist"
Oct 30 21:35:44.440872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.440838    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"]
Oct 30 21:35:44.449182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.449149    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3afc5e-857c9d96b8-4lrgv"]
Oct 30 21:35:44.498630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.497747    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"]
Oct 30 21:35:44.515675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:44.514348    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3afc5e-59dc9ffc9f-v4dhp"]
Oct 30 21:35:45.214864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:45.214833    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5374b820-82f7-47ab-a547-ea7a2563331c" path="/var/lib/kubelet/pods/5374b820-82f7-47ab-a547-ea7a2563331c/volumes"
Oct 30 21:35:45.215293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:45.215279    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b937cc-0b95-4612-859f-7cb9bcc4e5b1" path="/var/lib/kubelet/pods/75b937cc-0b95-4612-859f-7cb9bcc4e5b1/volumes"
Oct 30 21:35:45.215667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:45.215654    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d5fce9-d075-4ece-849a-8212323a51a2" path="/var/lib/kubelet/pods/78d5fce9-d075-4ece-849a-8212323a51a2/volumes"
Oct 30 21:35:46.260060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.259998    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:35:46.388354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.388322    6695 generic.go:356] "Generic (PLEG): container finished" podID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerID="158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6" exitCode=0
Oct 30 21:35:46.388507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.388378    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" event={"ID":"367c979a-ab4b-4145-af58-dfb2df6c1833","Type":"ContainerDied","Data":"158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6"}
Oct 30 21:35:46.388507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.388416    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk" event={"ID":"367c979a-ab4b-4145-af58-dfb2df6c1833","Type":"ContainerDied","Data":"f93372b6166dbf10d006a1f8f0b943296b952603279b8b342bd0ba31927af238"}
Oct 30 21:35:46.388507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.388437    6695 scope.go:122] "RemoveContainer" containerID="158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6"
Oct 30 21:35:46.388507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.388380    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"
Oct 30 21:35:46.398963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.398942    6695 scope.go:122] "RemoveContainer" containerID="158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6"
Oct 30 21:35:46.399282 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:35:46.399257    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6\": container with ID starting with 158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6 not found: ID does not exist" containerID="158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6"
Oct 30 21:35:46.399393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.399286    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6"} err="failed to get container status \"158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6\": rpc error: code = NotFound desc = could not find container \"158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6\": container with ID starting with 158712276ea0f5aed21883e60b08b280bcc213e06edf64989b94d15557dc1ab6 not found: ID does not exist"
Oct 30 21:35:46.401699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401680    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.401797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401785    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.401849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401809    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.401903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401846    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.401903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401877    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.402012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401934    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.402012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.401964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.402128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402094    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.402189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402146    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:46.402189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402173    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket\") pod \"367c979a-ab4b-4145-af58-dfb2df6c1833\" (UID: \"367c979a-ab4b-4145-af58-dfb2df6c1833\") "
Oct 30 21:35:46.402389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402203    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:46.402389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402371    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:46.402525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402501    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:46.402616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402581    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:46.402880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402847    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.402880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402873    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.402999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402888    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.402999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402900    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.402999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.402913    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/367c979a-ab4b-4145-af58-dfb2df6c1833-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.404141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.404119    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:35:46.404259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.404239    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:46.404534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.404511    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "kube-api-access-9nbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:46.404534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.404520    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy" pod "367c979a-ab4b-4145-af58-dfb2df6c1833" (UID: "367c979a-ab4b-4145-af58-dfb2df6c1833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:46.504131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.504093    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nbbv\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-kube-api-access-9nbbv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.504131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.504136    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.504335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.504148    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.504335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.504157    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/367c979a-ab4b-4145-af58-dfb2df6c1833-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:46.742484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.742450    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"]
Oct 30 21:35:46.753198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:46.753168    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-jkmxk"]
Oct 30 21:35:47.214835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:47.214803    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" path="/var/lib/kubelet/pods/367c979a-ab4b-4145-af58-dfb2df6c1833/volumes"
Oct 30 21:35:54.666745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:54.666707    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"]
Oct 30 21:35:54.667205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:54.666944    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy" containerID="cri-o://b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d" gracePeriod=30
Oct 30 21:35:55.254535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.254499    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"]
Oct 30 21:35:55.255143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255126    6695 cpu_manager.go:414] "containerMap: removing container" podUID="78d5fce9-d075-4ece-849a-8212323a51a2" containerName="payload-pre-processing"
Oct 30 21:35:55.255191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255145    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d5fce9-d075-4ece-849a-8212323a51a2" containerName="payload-pre-processing"
Oct 30 21:35:55.255191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255188    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5374b820-82f7-47ab-a547-ea7a2563331c" containerName="payload-processing"
Oct 30 21:35:55.255259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255194    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5374b820-82f7-47ab-a547-ea7a2563331c" containerName="payload-processing"
Oct 30 21:35:55.255259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255203    6695 cpu_manager.go:414] "containerMap: removing container" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy"
Oct 30 21:35:55.255259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255208    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy"
Oct 30 21:35:55.255367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255289    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d5fce9-d075-4ece-849a-8212323a51a2" containerName="payload-pre-processing"
Oct 30 21:35:55.255367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255305    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5374b820-82f7-47ab-a547-ea7a2563331c" containerName="payload-processing"
Oct 30 21:35:55.255367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.255311    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="367c979a-ab4b-4145-af58-dfb2df6c1833" containerName="istio-proxy"
Oct 30 21:35:55.259543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.259528    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.279016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.278993    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:35:55.279173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.279043    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:35:55.290562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.290532    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:35:55.290745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.290729    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:35:55.293651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.293628    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"]
Oct 30 21:35:55.378114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.378079    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.378328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.378311    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.378547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.378529    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4d82\" (UniqueName: \"kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.480411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.480373    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4d82\" (UniqueName: \"kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.480584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.480469    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.480584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.480522    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.481229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.481204    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.482983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.482962    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.503244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.503220    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4d82\" (UniqueName: \"kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82\") pod \"maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.573769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.573736    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:55.892615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:55.892530    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"]
Oct 30 21:35:55.896586 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:35:55.896551    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5ee85a_89c2_4a20_b3e8_4a1186dc6f77.slice/crio-2cb6376036bc272a7dde74b2920506e863ff5bf480bd5e2b623cb74555661481 WatchSource:0}: Error finding container 2cb6376036bc272a7dde74b2920506e863ff5bf480bd5e2b623cb74555661481: Status 404 returned error can't find the container with id 2cb6376036bc272a7dde74b2920506e863ff5bf480bd5e2b623cb74555661481
Oct 30 21:35:56.470277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:56.470166    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns" event={"ID":"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77","Type":"ContainerStarted","Data":"a5f544868ab2255f93ef401309ae98b7762f52f5188336806deadab083f7e205"}
Oct 30 21:35:56.470277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:56.470240    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns" event={"ID":"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77","Type":"ContainerStarted","Data":"2cb6376036bc272a7dde74b2920506e863ff5bf480bd5e2b623cb74555661481"}
Oct 30 21:35:56.723966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:56.723846    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns" podStartSLOduration=1.723818843 podStartE2EDuration="1.723818843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:56.709663403 +0000 UTC m=+1539.632121702" watchObservedRunningTime="2026-10-30 21:35:56.723818843 +0000 UTC m=+1539.646277142"
Oct 30 21:35:57.479199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:57.479165    6695 generic.go:356] "Generic (PLEG): container finished" podID="6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" containerID="a5f544868ab2255f93ef401309ae98b7762f52f5188336806deadab083f7e205" exitCode=0
Oct 30 21:35:57.479583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:57.479257    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns" event={"ID":"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77","Type":"ContainerDied","Data":"a5f544868ab2255f93ef401309ae98b7762f52f5188336806deadab083f7e205"}
Oct 30 21:35:58.648432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.648400    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:35:58.724436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.724397    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82\" (UniqueName: \"kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82\") pod \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") "
Oct 30 21:35:58.724651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.724486    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca\") pod \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") "
Oct 30 21:35:58.724651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.724558    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca\") pod \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\" (UID: \"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77\") "
Oct 30 21:35:58.724990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.724956    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca" pod "6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" (UID: "6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:58.725261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.725239    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:58.727292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.727262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82" pod "6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" (UID: "6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77"). InnerVolumeSpecName "kube-api-access-t4d82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:58.728046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.728015    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca" pod "6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" (UID: "6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:58.826762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.826725    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4d82\" (UniqueName: \"kubernetes.io/projected/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-kube-api-access-t4d82\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:58.826762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:58.826759    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:35:59.496475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:59.496385    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns" event={"ID":"6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77","Type":"ContainerDied","Data":"2cb6376036bc272a7dde74b2920506e863ff5bf480bd5e2b623cb74555661481"}
Oct 30 21:35:59.496771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:59.496740    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cb6376036bc272a7dde74b2920506e863ff5bf480bd5e2b623cb74555661481"
Oct 30 21:35:59.496771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:35:59.496437    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"
Oct 30 21:36:00.007346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.007316    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:36:00.067273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067191    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067314    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067386    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067441    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067511    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067548    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067621    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067660    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:00.067833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067686    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067726    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert\") pod \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\" (UID: \"6259c498-a1b5-4aba-8d29-1fe6a590d24f\") "
Oct 30 21:36:00.067985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067670    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:00.067985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067822    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:00.067985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.067894    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:00.068251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.068222    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:00.068404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.068349    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.068404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.068369    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.068404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.068384    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.068404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.068398    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.068707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.068413    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.069819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.069789    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:36:00.069934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.069854    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:00.069984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.069950    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:00.070097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.070082    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw" pod "6259c498-a1b5-4aba-8d29-1fe6a590d24f" (UID: "6259c498-a1b5-4aba-8d29-1fe6a590d24f"). InnerVolumeSpecName "kube-api-access-ffgtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:00.169464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.169420    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.169464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.169454    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.169464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.169468    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6259c498-a1b5-4aba-8d29-1fe6a590d24f-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.169723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.169481    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgtw\" (UniqueName: \"kubernetes.io/projected/6259c498-a1b5-4aba-8d29-1fe6a590d24f-kube-api-access-ffgtw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:00.505567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.505478    6695 generic.go:356] "Generic (PLEG): container finished" podID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerID="b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d" exitCode=0
Oct 30 21:36:00.505567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.505554    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"
Oct 30 21:36:00.505792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.505558    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" event={"ID":"6259c498-a1b5-4aba-8d29-1fe6a590d24f","Type":"ContainerDied","Data":"b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d"}
Oct 30 21:36:00.505792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.505653    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf" event={"ID":"6259c498-a1b5-4aba-8d29-1fe6a590d24f","Type":"ContainerDied","Data":"1af5f1b281aab579508991334f03b39498646c53dd40f2ae2a486817f2fc3b3e"}
Oct 30 21:36:00.505792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.505677    6695 scope.go:122] "RemoveContainer" containerID="b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d"
Oct 30 21:36:00.516235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.516217    6695 scope.go:122] "RemoveContainer" containerID="3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"
Oct 30 21:36:00.526333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.526314    6695 scope.go:122] "RemoveContainer" containerID="b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d"
Oct 30 21:36:00.526617 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:00.526582    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d\": container with ID starting with b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d not found: ID does not exist" containerID="b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d"
Oct 30 21:36:00.526679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.526630    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d"} err="failed to get container status \"b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d\": rpc error: code = NotFound desc = could not find container \"b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d\": container with ID starting with b5fe0d00c25b1bbda028be124b366e3288cb54508ed1f7551d9562339fd81f6d not found: ID does not exist"
Oct 30 21:36:00.526679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.526649    6695 scope.go:122] "RemoveContainer" containerID="3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"
Oct 30 21:36:00.526907 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:00.526887    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9\": container with ID starting with 3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9 not found: ID does not exist" containerID="3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"
Oct 30 21:36:00.526949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.526914    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9"} err="failed to get container status \"3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9\": rpc error: code = NotFound desc = could not find container \"3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9\": container with ID starting with 3e29418e8d093865ed7ef480fb57be8cf0e2f240e34a4e9b982fb2930a37d9c9 not found: ID does not exist"
Oct 30 21:36:00.581458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.581417    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"]
Oct 30 21:36:00.593007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:00.592959    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3afc5e-openshift-default-f958f8cd9-2s8tf"]
Oct 30 21:36:01.075759 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:01.075722    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:36:01.214783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:01.214751    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" path="/var/lib/kubelet/pods/6259c498-a1b5-4aba-8d29-1fe6a590d24f/volumes"
Oct 30 21:36:02.475277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.475240    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"]
Oct 30 21:36:02.476191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476164    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy"
Oct 30 21:36:02.476281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476193    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy"
Oct 30 21:36:02.476281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476233    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy"
Oct 30 21:36:02.476281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476241    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy"
Oct 30 21:36:02.476281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476266    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" containerName="revoke-keys"
Oct 30 21:36:02.476281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476274    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" containerName="revoke-keys"
Oct 30 21:36:02.476545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476458    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" containerName="revoke-keys"
Oct 30 21:36:02.476545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476484    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy"
Oct 30 21:36:02.476545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.476495    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6259c498-a1b5-4aba-8d29-1fe6a590d24f" containerName="istio-proxy"
Oct 30 21:36:02.482137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.482113    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.505141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.505100    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f05cda-openshift-default-dockercfg-p2fkc\""
Oct 30 21:36:02.519355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.519322    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"]
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.616941    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617054    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617126    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617148    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617171    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4h2\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617196    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617380    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617433    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.623625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.617519    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719068    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719157    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719185    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719209    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4h2\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719355    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719396    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719475    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.719526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.719519    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.720115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.720084    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.720390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.720367    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.720490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.720411    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.720490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.720421    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.720700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.720675    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.723000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.722969    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.728445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.728373    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.752074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.752043    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.752335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.752310    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4h2\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2\") pod \"e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") " pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:02.797506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:02.797463    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:03.182713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.182679    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"]
Oct 30 21:36:03.182957 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:03.182931    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221369ca_f9e3_4d12_8279_13e85182e7c9.slice/crio-e16ac5536c79278d03c8cda756f05542e767c848ccff55f176899238a808b6bd WatchSource:0}: Error finding container e16ac5536c79278d03c8cda756f05542e767c848ccff55f176899238a808b6bd: Status 404 returned error can't find the container with id e16ac5536c79278d03c8cda756f05542e767c848ccff55f176899238a808b6bd
Oct 30 21:36:03.185710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.185671    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:03.185824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.185766    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:03.185883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.185824    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:03.533861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.533827    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" event={"ID":"221369ca-f9e3-4d12-8279-13e85182e7c9","Type":"ContainerStarted","Data":"16d05cc93d83599bd5c8430a943dc4423ab0a7f682e609c55428ce0131f82d0d"}
Oct 30 21:36:03.534265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.533876    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" event={"ID":"221369ca-f9e3-4d12-8279-13e85182e7c9","Type":"ContainerStarted","Data":"e16ac5536c79278d03c8cda756f05542e767c848ccff55f176899238a808b6bd"}
Oct 30 21:36:03.626991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.626924    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podStartSLOduration=1.626904182 podStartE2EDuration="1.626904182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:03.621581238 +0000 UTC m=+1546.544039535" watchObservedRunningTime="2026-10-30 21:36:03.626904182 +0000 UTC m=+1546.549362480"
Oct 30 21:36:03.797814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:03.797711    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:04.804067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:04.804037    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:05.547524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:05.547490    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:05.548470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:05.548449    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:36:08.751636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.749179    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"]
Oct 30 21:36:08.756952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.756910    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:08.759023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.757656    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"]
Oct 30 21:36:08.762274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.762239    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"]
Oct 30 21:36:08.787954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.786468    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f05cda\""
Oct 30 21:36:08.788919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.788893    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"]
Oct 30 21:36:08.789191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.789136    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:08.840629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.839130    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f05cda-dockercfg-8qxwb\""
Oct 30 21:36:08.905985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.905946    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:08.993829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.993798    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmw42\" (UniqueName: \"kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42\") pod \"payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:08.994641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.994595    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq85n\" (UniqueName: \"kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n\") pod \"payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:08.995081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:08.995059    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:09.097485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.097406    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:09.097721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.097567    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:09.097721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.097657    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmw42\" (UniqueName: \"kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42\") pod \"payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:09.097721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.097700    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq85n\" (UniqueName: \"kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n\") pod \"payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:09.098433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.098402    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:09.098635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.098484    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:09.143628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.143503    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq85n\" (UniqueName: \"kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n\") pod \"payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:09.143628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.143522    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmw42\" (UniqueName: \"kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42\") pod \"payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:09.384473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.384376    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:09.429643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.428872    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:09.955690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:09.955655    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"]
Oct 30 21:36:09.959435 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:09.959404    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26a5b26_e3bd_4250_974a_86400742b17b.slice/crio-a6132fe2619fae342e849c0494a9eb0c2d31a776a8579806b992958eff40bbff WatchSource:0}: Error finding container a6132fe2619fae342e849c0494a9eb0c2d31a776a8579806b992958eff40bbff: Status 404 returned error can't find the container with id a6132fe2619fae342e849c0494a9eb0c2d31a776a8579806b992958eff40bbff
Oct 30 21:36:10.007848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.007822    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"]
Oct 30 21:36:10.707437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.707400    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm" event={"ID":"d26a5b26-e3bd-4250-974a-86400742b17b","Type":"ContainerStarted","Data":"993ca4b8e70a02b38d62e78f0dcfd27e273b011be876476e846573d9f47fc7b5"}
Oct 30 21:36:10.707665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.707472    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm" event={"ID":"d26a5b26-e3bd-4250-974a-86400742b17b","Type":"ContainerStarted","Data":"a6132fe2619fae342e849c0494a9eb0c2d31a776a8579806b992958eff40bbff"}
Oct 30 21:36:10.707747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.707665    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:10.709366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.709341    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c" event={"ID":"ab850567-4e74-48e9-9ff5-14795d4e0bc5","Type":"ContainerStarted","Data":"056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989"}
Oct 30 21:36:10.709469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.709379    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c" event={"ID":"ab850567-4e74-48e9-9ff5-14795d4e0bc5","Type":"ContainerStarted","Data":"82572cb83aa712b2f0729c483078e151fb3a14765834b0390a3cf1265dd13f5c"}
Oct 30 21:36:10.709469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.709398    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:10.769014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.768948    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm" podStartSLOduration=2.7689299370000002 podStartE2EDuration="2.768929937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:10.768498509 +0000 UTC m=+1553.690956809" watchObservedRunningTime="2026-10-30 21:36:10.768929937 +0000 UTC m=+1553.691388236"
Oct 30 21:36:10.839807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:10.839743    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c" podStartSLOduration=2.839726999 podStartE2EDuration="2.839726999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:10.839348362 +0000 UTC m=+1553.761806687" watchObservedRunningTime="2026-10-30 21:36:10.839726999 +0000 UTC m=+1553.762185298"
Oct 30 21:36:16.583688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.583648    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"]
Oct 30 21:36:16.589270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.589243    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.632625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.632202    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ea45c0-openshift-default-dockercfg-dxtx9\""
Oct 30 21:36:16.639171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.639137    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"]
Oct 30 21:36:16.676729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.676672    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.676951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.676786    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.676951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.676825    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.676951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.676865    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.676951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.676896    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.676951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.676945    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.677246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.677019    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.677246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.677061    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.677246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.677134    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8b9\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787584    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787701    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787733    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787763    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787794    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787840    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787872    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787912    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.787948    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8b9\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.788778    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.789294    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.789865    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.790090    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.795622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.792912    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.841114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.829350    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8b9\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.841114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.832308    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.864633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.851634    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.901068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.899678    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token\") pod \"e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") " pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:16.910615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:16.906902    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:17.370897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.370850    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"]
Oct 30 21:36:17.375802 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:17.375767    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc482d66f_1a6c_4407_8c32_20b4fb649d28.slice/crio-daf6d990e7992fd490ae48725bc1f2ef2b8f52f19784d6f4583e4e9068da4897 WatchSource:0}: Error finding container daf6d990e7992fd490ae48725bc1f2ef2b8f52f19784d6f4583e4e9068da4897: Status 404 returned error can't find the container with id daf6d990e7992fd490ae48725bc1f2ef2b8f52f19784d6f4583e4e9068da4897
Oct 30 21:36:17.378996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.378947    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:17.379106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.379052    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:17.379147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.379132    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:17.897386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.897337    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" event={"ID":"c482d66f-1a6c-4407-8c32-20b4fb649d28","Type":"ContainerStarted","Data":"2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"}
Oct 30 21:36:17.897900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.897398    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" event={"ID":"c482d66f-1a6c-4407-8c32-20b4fb649d28","Type":"ContainerStarted","Data":"daf6d990e7992fd490ae48725bc1f2ef2b8f52f19784d6f4583e4e9068da4897"}
Oct 30 21:36:17.908171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:17.908139    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:18.860776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:18.860745    6695 scope.go:122] "RemoveContainer" containerID="0015ac98f5ceeecdfdbec5b362e65a1e8f29f3c83118d158eb23d2dd31a07383"
Oct 30 21:36:18.872322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:18.871880    6695 scope.go:122] "RemoveContainer" containerID="197f917e2c71a0b222b5825d7e06b612e7aa81edcd9d4936c72ff037e5cc4b06"
Oct 30 21:36:18.888588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:18.888558    6695 scope.go:122] "RemoveContainer" containerID="8080418b1cedbe04cbdbecc795937a13cb835ce469948ac13900b7d3038bf24b"
Oct 30 21:36:18.939204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:18.938833    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:19.011385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.011319    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podStartSLOduration=3.011297822 podStartE2EDuration="3.011297822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:17.985899216 +0000 UTC m=+1560.908357513" watchObservedRunningTime="2026-10-30 21:36:19.011297822 +0000 UTC m=+1561.933756121"
Oct 30 21:36:19.938177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.938141    6695 generic.go:356] "Generic (PLEG): container finished" podID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerID="546ce07ba3010e65fd9fb89b79cf3cc4f930ae8801c4f0c7a61dac5535f84f4e" exitCode=0
Oct 30 21:36:19.938421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.938219    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" event={"ID":"37ac6495-1f88-4207-ad29-92fd7e2aba41","Type":"ContainerDied","Data":"546ce07ba3010e65fd9fb89b79cf3cc4f930ae8801c4f0c7a61dac5535f84f4e"}
Oct 30 21:36:19.938755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.938725    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:19.938890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.938869    6695 scope.go:122] "RemoveContainer" containerID="546ce07ba3010e65fd9fb89b79cf3cc4f930ae8801c4f0c7a61dac5535f84f4e"
Oct 30 21:36:19.939432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.939410    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:36:19.940836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.940802    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:19.940947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.940926    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:19.940993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:19.940979    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:20.695983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.681516    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"]
Oct 30 21:36:20.695983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.685634    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.734630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.732646    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-8a20c6-openshift-default-dockercfg-9c7tp\""
Oct 30 21:36:20.743592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.743556    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"]
Oct 30 21:36:20.766337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766303    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766349    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766616    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766663    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mds7s\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766714    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766756    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766782    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766868    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:20.766948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:20.766913    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.041623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.033483    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.050678    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.050753    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds7s\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.050813    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.050860    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.050888    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.050984    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.051033    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.051132    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.051167    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.051812    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.052381    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.052635    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.063704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.053152    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.092672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.089625    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.092672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.090682    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" event={"ID":"37ac6495-1f88-4207-ad29-92fd7e2aba41","Type":"ContainerStarted","Data":"b31ad7054105ed753e3c68e43da0ec6706b3f0673d7a4a46f993fd8a013e6a94"}
Oct 30 21:36:21.092672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.091888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.097027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.096989    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds7s\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.097027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.097007    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token\") pod \"e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") " pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.139631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.138714    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:36:21.302186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.302091    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:21.536084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.536048    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:36:21.537279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.537106    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 21:36:21.537279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.537185    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 21:36:21.672499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.672470    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"]
Oct 30 21:36:21.674507 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:21.674480    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c1f544_3c15_41fe_aed1_17449c0e7766.slice/crio-de78553c7326d430b57735fb3c62bad70a76b272e2fa85c34d803f7318effbe6 WatchSource:0}: Error finding container de78553c7326d430b57735fb3c62bad70a76b272e2fa85c34d803f7318effbe6: Status 404 returned error can't find the container with id de78553c7326d430b57735fb3c62bad70a76b272e2fa85c34d803f7318effbe6
Oct 30 21:36:21.677183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.677151    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:21.677271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.677218    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:21.677312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.677270    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:21.739914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.739762    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:36:21.740246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:21.740226    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:36:22.103284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.103255    6695 generic.go:356] "Generic (PLEG): container finished" podID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerID="acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc" exitCode=0
Oct 30 21:36:22.104054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.103809    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" event={"ID":"b49aa851-f301-40f5-bc59-cc2fb9b80ccf","Type":"ContainerDied","Data":"acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"}
Oct 30 21:36:22.104466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.104441    6695 scope.go:122] "RemoveContainer" containerID="acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"
Oct 30 21:36:22.106633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.106573    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" event={"ID":"f7c1f544-3c15-41fe-aed1-17449c0e7766","Type":"ContainerStarted","Data":"f615c346ad4f1deba3c7580da28aba46133d6e7af342e49a070444bd36274b5c"}
Oct 30 21:36:22.106756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.106644    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" event={"ID":"f7c1f544-3c15-41fe-aed1-17449c0e7766","Type":"ContainerStarted","Data":"de78553c7326d430b57735fb3c62bad70a76b272e2fa85c34d803f7318effbe6"}
Oct 30 21:36:22.107002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.106970    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:22.107371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.107057    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:22.107371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.107100    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:22.249297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.249237    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podStartSLOduration=2.249218715 podStartE2EDuration="2.249218715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:22.243517552 +0000 UTC m=+1565.165975862" watchObservedRunningTime="2026-10-30 21:36:22.249218715 +0000 UTC m=+1565.171677014"
Oct 30 21:36:22.302596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.302563    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:22.329684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.310336    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:22.545579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.545538    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 21:36:22.545838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.545639    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 21:36:22.741730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.740016    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"]
Oct 30 21:36:22.744077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.743324    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:22.760160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.760134    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"]
Oct 30 21:36:22.762029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.762010    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ea45c0\""
Oct 30 21:36:22.771671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.771649    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ea45c0-dockercfg-ssdrp\""
Oct 30 21:36:22.775953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.775926    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"]
Oct 30 21:36:22.788530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.788493    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmfjm\" (UniqueName: \"kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm\") pod \"payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:22.871616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.840581    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:22.887205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:22.887170    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:23.048247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.048191    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmfjm\" (UniqueName: \"kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm\") pod \"payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:23.048625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.048584    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:23.048948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.048913    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcblw\" (UniqueName: \"kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw\") pod \"payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.049112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.049092    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.050541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.050490    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:23.057627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.057581    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"]
Oct 30 21:36:23.083354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.083291    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmfjm\" (UniqueName: \"kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm\") pod \"payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:23.151663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.138663    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" event={"ID":"b49aa851-f301-40f5-bc59-cc2fb9b80ccf","Type":"ContainerStarted","Data":"da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec"}
Oct 30 21:36:23.151663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.138761    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:23.151663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.140795    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:36:23.154677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.154642    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcblw\" (UniqueName: \"kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw\") pod \"payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.154931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.154895    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.155590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.155567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.193294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.193241    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcblw\" (UniqueName: \"kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw\") pod \"payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.355988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.355885    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:23.359974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.359859    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:23.557895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.557383    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 21:36:23.557895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.557451    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 21:36:23.565676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.560114    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:36:23.565676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.560159    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 21:36:23.565676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.560222    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:36:23.767937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.767909    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"]
Oct 30 21:36:23.770411 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:23.770383    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff444b02_6644_4b46_8872_d2c0f8de1afc.slice/crio-f0466583e8a6bf1d38ab8b82b040a0d9e55521b62f014f8253be3a28695d0415 WatchSource:0}: Error finding container f0466583e8a6bf1d38ab8b82b040a0d9e55521b62f014f8253be3a28695d0415: Status 404 returned error can't find the container with id f0466583e8a6bf1d38ab8b82b040a0d9e55521b62f014f8253be3a28695d0415
Oct 30 21:36:23.774563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:23.774535    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"]
Oct 30 21:36:23.785323 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:23.785293    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544ae917_21bc_4dac_9db3_218298b99122.slice/crio-677192cf5ce034e06e1582b6d98dab5bc9810e20371dbd129ad08b86ae993e7c WatchSource:0}: Error finding container 677192cf5ce034e06e1582b6d98dab5bc9810e20371dbd129ad08b86ae993e7c: Status 404 returned error can't find the container with id 677192cf5ce034e06e1582b6d98dab5bc9810e20371dbd129ad08b86ae993e7c
Oct 30 21:36:24.170658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.169828    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6" event={"ID":"544ae917-21bc-4dac-9db3-218298b99122","Type":"ContainerStarted","Data":"b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5"}
Oct 30 21:36:24.170658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.169933    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6" event={"ID":"544ae917-21bc-4dac-9db3-218298b99122","Type":"ContainerStarted","Data":"677192cf5ce034e06e1582b6d98dab5bc9810e20371dbd129ad08b86ae993e7c"}
Oct 30 21:36:24.171132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.171025    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:24.173211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.173181    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd" event={"ID":"ff444b02-6644-4b46-8872-d2c0f8de1afc","Type":"ContainerStarted","Data":"4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a"}
Oct 30 21:36:24.173416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.173401    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:24.173535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.173520    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd" event={"ID":"ff444b02-6644-4b46-8872-d2c0f8de1afc","Type":"ContainerStarted","Data":"f0466583e8a6bf1d38ab8b82b040a0d9e55521b62f014f8253be3a28695d0415"}
Oct 30 21:36:24.233629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.229622    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6" podStartSLOduration=2.229579342 podStartE2EDuration="2.229579342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:24.228882776 +0000 UTC m=+1567.151341075" watchObservedRunningTime="2026-10-30 21:36:24.229579342 +0000 UTC m=+1567.152037643"
Oct 30 21:36:24.540627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.531016    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 21:36:24.540627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.531085    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 21:36:24.566681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.553768    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:36:24.566681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:24.553826    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 21:36:25.549627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:25.537714    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 21:36:25.549627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:25.537781    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 21:36:25.562627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:25.553982    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:36:25.562627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:25.554045    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 21:36:26.545761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:26.545716    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:36:26.545969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:26.545786    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 21:36:26.668955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:26.668921    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:36:26.670326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:26.670306    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:36:26.766243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:26.766186    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd" podStartSLOduration=4.76617326 podStartE2EDuration="4.76617326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:24.302632148 +0000 UTC m=+1567.225090437" watchObservedRunningTime="2026-10-30 21:36:26.76617326 +0000 UTC m=+1569.688631557"
Oct 30 21:36:27.240917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.240882    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"]
Oct 30 21:36:27.241142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.241101    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy" containerID="cri-o://57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c" gracePeriod=30
Oct 30 21:36:27.546005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.545967    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:36:27.546232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.546033    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 21:36:27.949760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.949673    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"]
Oct 30 21:36:27.953356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.953324    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:27.977082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.977058    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-8a20c6-dockercfg-2bpgw\""
Oct 30 21:36:27.977585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.977563    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-8a20c6\""
Oct 30 21:36:27.991668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.991633    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:27.992131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.991686    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vqw\" (UniqueName: \"kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw\") pod \"payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:27.993828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:27.993803    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"]
Oct 30 21:36:28.089625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.076975    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"]
Oct 30 21:36:28.089625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.084774    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.104633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.098278    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:28.104633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.098333    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vqw\" (UniqueName: \"kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw\") pod \"payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:28.104633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.099754    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:28.303529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.303492    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vqw\" (UniqueName: \"kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw\") pod \"payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:28.305494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.305460    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jrc\" (UniqueName: \"kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc\") pod \"payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2\" (UID: \"7283c843-f443-427c-ac30-825416123201\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.305672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.305570    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2\" (UID: \"7283c843-f443-427c-ac30-825416123201\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.348150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.347878    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:28.348341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.348169    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"]
Oct 30 21:36:28.407921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.407877    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jrc\" (UniqueName: \"kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc\") pod \"payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2\" (UID: \"7283c843-f443-427c-ac30-825416123201\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.408088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.408015    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2\" (UID: \"7283c843-f443-427c-ac30-825416123201\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.437707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.409662    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2\" (UID: \"7283c843-f443-427c-ac30-825416123201\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.440718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.440640    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jrc\" (UniqueName: \"kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc\") pod \"payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2\" (UID: \"7283c843-f443-427c-ac30-825416123201\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.472084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.472028    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:28.548632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.546636    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:36:28.548632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.546718    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 21:36:28.678904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.678868    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"]
Oct 30 21:36:28.682988 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:28.682961    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfef07d_e40c_4069_8288_932dfc99f368.slice/crio-41fcca3b4895fc8e75ce6411eeafd603e2a7369d6bb2ed6d79c23f01c2e0c396 WatchSource:0}: Error finding container 41fcca3b4895fc8e75ce6411eeafd603e2a7369d6bb2ed6d79c23f01c2e0c396: Status 404 returned error can't find the container with id 41fcca3b4895fc8e75ce6411eeafd603e2a7369d6bb2ed6d79c23f01c2e0c396
Oct 30 21:36:28.822543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:28.822318    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"]
Oct 30 21:36:28.826586 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:28.826556    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7283c843_f443_427c_ac30_825416123201.slice/crio-dc05a386694e6c6e9d9647a7c4100973a83026a12fc16fac5e4982a7ea659cef WatchSource:0}: Error finding container dc05a386694e6c6e9d9647a7c4100973a83026a12fc16fac5e4982a7ea659cef: Status 404 returned error can't find the container with id dc05a386694e6c6e9d9647a7c4100973a83026a12fc16fac5e4982a7ea659cef
Oct 30 21:36:29.324614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.324576    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h" event={"ID":"3bfef07d-e40c-4069-8288-932dfc99f368","Type":"ContainerStarted","Data":"9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63"}
Oct 30 21:36:29.325098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.324640    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h" event={"ID":"3bfef07d-e40c-4069-8288-932dfc99f368","Type":"ContainerStarted","Data":"41fcca3b4895fc8e75ce6411eeafd603e2a7369d6bb2ed6d79c23f01c2e0c396"}
Oct 30 21:36:29.325098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.324829    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:29.326353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.326324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2" event={"ID":"7283c843-f443-427c-ac30-825416123201","Type":"ContainerStarted","Data":"245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23"}
Oct 30 21:36:29.326478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.326366    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2" event={"ID":"7283c843-f443-427c-ac30-825416123201","Type":"ContainerStarted","Data":"dc05a386694e6c6e9d9647a7c4100973a83026a12fc16fac5e4982a7ea659cef"}
Oct 30 21:36:29.326478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.326394    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:29.372526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.372474    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h" podStartSLOduration=2.3724604080000002 podStartE2EDuration="2.372460408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:29.371003467 +0000 UTC m=+1572.293461766" watchObservedRunningTime="2026-10-30 21:36:29.372460408 +0000 UTC m=+1572.294918706"
Oct 30 21:36:29.416715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.416656    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2" podStartSLOduration=1.416641436 podStartE2EDuration="1.416641436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:29.41372715 +0000 UTC m=+1572.336185472" watchObservedRunningTime="2026-10-30 21:36:29.416641436 +0000 UTC m=+1572.339099761"
Oct 30 21:36:29.600091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.599807    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:36:29.605714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.601754    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:36:29.609949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:29.609920    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:36:31.074023 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:31.073987    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:36:32.514682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.514655    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:36:32.597409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597363    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597465    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597526    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597626    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597705    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597780    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597875    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.597932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597881    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:32.597932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597908    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.598153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.597969    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data\") pod \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\" (UID: \"404cec42-23cb-4e3c-ae29-fb05c11f1da7\") "
Oct 30 21:36:32.598210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598161    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.598313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598236    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.598469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598428    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.598863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598841    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.598983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598869    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.598983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598884    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.598983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.598900    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.601032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.599803    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.601032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.600098    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:36:32.601032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.600332    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:32.601383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.601361    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.601649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.601596    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52" pod "404cec42-23cb-4e3c-ae29-fb05c11f1da7" (UID: "404cec42-23cb-4e3c-ae29-fb05c11f1da7"). InnerVolumeSpecName "kube-api-access-vpv52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:32.700412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.700373    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.700412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.700405    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.700412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.700415    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpv52\" (UniqueName: \"kubernetes.io/projected/404cec42-23cb-4e3c-ae29-fb05c11f1da7-kube-api-access-vpv52\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.700690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.700427    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.700690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:32.700437    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cec42-23cb-4e3c-ae29-fb05c11f1da7-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:33.355929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.355893    6695 generic.go:356] "Generic (PLEG): container finished" podID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerID="57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c" exitCode=0
Oct 30 21:36:33.356145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.355950    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"
Oct 30 21:36:33.356145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.355981    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" event={"ID":"404cec42-23cb-4e3c-ae29-fb05c11f1da7","Type":"ContainerDied","Data":"57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c"}
Oct 30 21:36:33.356145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.356050    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x" event={"ID":"404cec42-23cb-4e3c-ae29-fb05c11f1da7","Type":"ContainerDied","Data":"f8926475c250e7440672bc056ad69efbb4fb38e0ad7a5016447b98446b74e968"}
Oct 30 21:36:33.356145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.356081    6695 scope.go:122] "RemoveContainer" containerID="57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c"
Oct 30 21:36:33.366828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.366689    6695 scope.go:122] "RemoveContainer" containerID="57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c"
Oct 30 21:36:33.367048 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:33.367021    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c\": container with ID starting with 57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c not found: ID does not exist" containerID="57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c"
Oct 30 21:36:33.367093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.367059    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c"} err="failed to get container status \"57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c\": rpc error: code = NotFound desc = could not find container \"57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c\": container with ID starting with 57a78c6bbe5629537bb5c9672d5c03f7213c2013e6d68ae27730a40ac291368c not found: ID does not exist"
Oct 30 21:36:33.423363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.423331    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"]
Oct 30 21:36:33.434005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:33.433966    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-bm55x"]
Oct 30 21:36:34.161406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.161371    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"]
Oct 30 21:36:34.161908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.161560    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79" podUID="bd2d7192-a9d1-4154-a69d-088b5ef30843" containerName="payload-processing" containerID="cri-o://c0e3af2c7a9cede1ee59491bc243a9437e3eac64a3fa7cdf40af7e2c1e5dac30" gracePeriod=30
Oct 30 21:36:34.201019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.200985    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"]
Oct 30 21:36:34.201231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.201195    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6" podUID="a6eef253-0b20-4a8f-827b-c7777bca0bb3" containerName="payload-pre-processing" containerID="cri-o://7e0b777b64074924dadd5bb3b9a95fd9484dab3e3c6dbaea28d5ddec4845ffdd" gracePeriod=30
Oct 30 21:36:34.369017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.368975    6695 generic.go:356] "Generic (PLEG): container finished" podID="a6eef253-0b20-4a8f-827b-c7777bca0bb3" containerID="7e0b777b64074924dadd5bb3b9a95fd9484dab3e3c6dbaea28d5ddec4845ffdd" exitCode=0
Oct 30 21:36:34.374444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.369648    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6" event={"ID":"a6eef253-0b20-4a8f-827b-c7777bca0bb3","Type":"ContainerDied","Data":"7e0b777b64074924dadd5bb3b9a95fd9484dab3e3c6dbaea28d5ddec4845ffdd"}
Oct 30 21:36:34.374444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.374053    6695 generic.go:356] "Generic (PLEG): container finished" podID="bd2d7192-a9d1-4154-a69d-088b5ef30843" containerID="c0e3af2c7a9cede1ee59491bc243a9437e3eac64a3fa7cdf40af7e2c1e5dac30" exitCode=0
Oct 30 21:36:34.374444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.374110    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79" event={"ID":"bd2d7192-a9d1-4154-a69d-088b5ef30843","Type":"ContainerDied","Data":"c0e3af2c7a9cede1ee59491bc243a9437e3eac64a3fa7cdf40af7e2c1e5dac30"}
Oct 30 21:36:34.622764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.622425    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:36:34.636271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.631752    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:36:34.729763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.729660    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn\" (UniqueName: \"kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn\") pod \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") "
Oct 30 21:36:34.729935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.729888    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume\") pod \"bd2d7192-a9d1-4154-a69d-088b5ef30843\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") "
Oct 30 21:36:34.729995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.729982    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn\" (UniqueName: \"kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn\") pod \"bd2d7192-a9d1-4154-a69d-088b5ef30843\" (UID: \"bd2d7192-a9d1-4154-a69d-088b5ef30843\") "
Oct 30 21:36:34.730049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.730033    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume\") pod \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\" (UID: \"a6eef253-0b20-4a8f-827b-c7777bca0bb3\") "
Oct 30 21:36:34.731044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.731014    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume" pod "a6eef253-0b20-4a8f-827b-c7777bca0bb3" (UID: "a6eef253-0b20-4a8f-827b-c7777bca0bb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:34.731201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.731076    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume" pod "bd2d7192-a9d1-4154-a69d-088b5ef30843" (UID: "bd2d7192-a9d1-4154-a69d-088b5ef30843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:34.734245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.734221    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn" pod "bd2d7192-a9d1-4154-a69d-088b5ef30843" (UID: "bd2d7192-a9d1-4154-a69d-088b5ef30843"). InnerVolumeSpecName "kube-api-access-s2rjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:34.736780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.736740    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn" pod "a6eef253-0b20-4a8f-827b-c7777bca0bb3" (UID: "a6eef253-0b20-4a8f-827b-c7777bca0bb3"). InnerVolumeSpecName "kube-api-access-bsnzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:34.831803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.831765    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnzn\" (UniqueName: \"kubernetes.io/projected/a6eef253-0b20-4a8f-827b-c7777bca0bb3-kube-api-access-bsnzn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.831803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.831800    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2d7192-a9d1-4154-a69d-088b5ef30843-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.831994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.831815    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2rjn\" (UniqueName: \"kubernetes.io/projected/bd2d7192-a9d1-4154-a69d-088b5ef30843-kube-api-access-s2rjn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.831994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:34.831829    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6eef253-0b20-4a8f-827b-c7777bca0bb3-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:35.070309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.070241    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"]
Oct 30 21:36:35.076053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.076006    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-38e5bf-8gtns"]
Oct 30 21:36:35.184781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.184746    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:36:35.215225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.215191    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" path="/var/lib/kubelet/pods/404cec42-23cb-4e3c-ae29-fb05c11f1da7/volumes"
Oct 30 21:36:35.215789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.215773    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77" path="/var/lib/kubelet/pods/6a5ee85a-89c2-4a20-b3e8-4a1186dc6f77/volumes"
Oct 30 21:36:35.389413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.389324    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"
Oct 30 21:36:35.389413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.389348    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79" event={"ID":"bd2d7192-a9d1-4154-a69d-088b5ef30843","Type":"ContainerDied","Data":"289108f303dbd34586f657c3a576ef566a80bac681d38fe13a873d5968f69950"}
Oct 30 21:36:35.389692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.389424    6695 scope.go:122] "RemoveContainer" containerID="c0e3af2c7a9cede1ee59491bc243a9437e3eac64a3fa7cdf40af7e2c1e5dac30"
Oct 30 21:36:35.391087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.391064    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6" event={"ID":"a6eef253-0b20-4a8f-827b-c7777bca0bb3","Type":"ContainerDied","Data":"829a85843e6e646e7870c91b641b91bf1db60541173dd701c345913468155ab9"}
Oct 30 21:36:35.391213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.391089    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"
Oct 30 21:36:35.400335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.400317    6695 scope.go:122] "RemoveContainer" containerID="7e0b777b64074924dadd5bb3b9a95fd9484dab3e3c6dbaea28d5ddec4845ffdd"
Oct 30 21:36:35.444067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.444036    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"]
Oct 30 21:36:35.462690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.462657    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-38e5bf-78cd77dcc4-bpz79"]
Oct 30 21:36:35.504627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.504579    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"]
Oct 30 21:36:35.512324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:35.512298    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-38e5bf-6d64486c79-kgtl6"]
Oct 30 21:36:36.189913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:36.189883    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:36:37.215192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:37.215159    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eef253-0b20-4a8f-827b-c7777bca0bb3" path="/var/lib/kubelet/pods/a6eef253-0b20-4a8f-827b-c7777bca0bb3/volumes"
Oct 30 21:36:37.215568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:37.215555    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2d7192-a9d1-4154-a69d-088b5ef30843" path="/var/lib/kubelet/pods/bd2d7192-a9d1-4154-a69d-088b5ef30843/volumes"
Oct 30 21:36:40.337203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:40.337171    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:36:40.337725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:40.337250    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:36:43.942952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:43.942917    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"]
Oct 30 21:36:43.943366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:43.943171    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" containerID="cri-o://ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5" gracePeriod=30
Oct 30 21:36:46.097582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:46.097548    6695 patch_prober.go:29] interesting pod/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:36:46.098211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:46.097628    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:36:47.365054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.365016    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"]
Oct 30 21:36:47.366721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.366691    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bd2d7192-a9d1-4154-a69d-088b5ef30843" containerName="payload-processing"
Oct 30 21:36:47.366924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.366907    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2d7192-a9d1-4154-a69d-088b5ef30843" containerName="payload-processing"
Oct 30 21:36:47.367072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.367060    6695 cpu_manager.go:414] "containerMap: removing container" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy"
Oct 30 21:36:47.367179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.367168    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy"
Oct 30 21:36:47.367391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.367376    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a6eef253-0b20-4a8f-827b-c7777bca0bb3" containerName="payload-pre-processing"
Oct 30 21:36:47.367517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.367502    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eef253-0b20-4a8f-827b-c7777bca0bb3" containerName="payload-pre-processing"
Oct 30 21:36:47.373800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.367810    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eef253-0b20-4a8f-827b-c7777bca0bb3" containerName="payload-pre-processing"
Oct 30 21:36:47.375034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.375009    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2d7192-a9d1-4154-a69d-088b5ef30843" containerName="payload-processing"
Oct 30 21:36:47.375212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.375198    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="404cec42-23cb-4e3c-ae29-fb05c11f1da7" containerName="istio-proxy"
Oct 30 21:36:47.379540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.379519    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.405804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.405776    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-912e61-openshift-default-dockercfg-hjhj7\""
Oct 30 21:36:47.408409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.408377    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"]
Oct 30 21:36:47.479411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479361    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.479640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479432    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jnsx\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.479640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479546    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.479789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479735    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.479789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479769    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.479914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479831    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.479914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479864    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.480035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.479948    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.480035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.480017    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581139    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581186    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581244    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581275    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581358    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581431    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581493    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581522    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jnsx\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581559    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581644    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.581711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581678    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.582008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.581991    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.582170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.582103    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.582170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.582110    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.583686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.583661    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.584154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.584133    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.601984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.601963    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.614119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.614093    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jnsx\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx\") pod \"e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") " pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:47.705226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:47.705122    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:48.049659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.049623    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"]
Oct 30 21:36:48.053361 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:48.053334    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd446414a_be00_4d2d_97cc_006645cb67d8.slice/crio-c4212339c1a77544f025a1eee43a10ba3340f052392cfe5bacb874becbae8580 WatchSource:0}: Error finding container c4212339c1a77544f025a1eee43a10ba3340f052392cfe5bacb874becbae8580: Status 404 returned error can't find the container with id c4212339c1a77544f025a1eee43a10ba3340f052392cfe5bacb874becbae8580
Oct 30 21:36:48.056139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.056109    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:48.056202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.056177    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:48.056239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.056207    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:48.562624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.561475    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" event={"ID":"d446414a-be00-4d2d-97cc-006645cb67d8","Type":"ContainerStarted","Data":"b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"}
Oct 30 21:36:48.562624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.561645    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" event={"ID":"d446414a-be00-4d2d-97cc-006645cb67d8","Type":"ContainerStarted","Data":"c4212339c1a77544f025a1eee43a10ba3340f052392cfe5bacb874becbae8580"}
Oct 30 21:36:48.644637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.644550    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podStartSLOduration=1.64453034 podStartE2EDuration="1.64453034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:48.629165374 +0000 UTC m=+1591.551623674" watchObservedRunningTime="2026-10-30 21:36:48.64453034 +0000 UTC m=+1591.566988641"
Oct 30 21:36:48.705440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:48.705401    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:49.043741 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:49.043698    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7dd11d_17cb_4ad5_b7ff_4341cfd9d0ba.slice/crio-conmon-ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:36:49.343394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.343367    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:36:49.433496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433449    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433517    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433569    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433631    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433662    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433745    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433781    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433798    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:49.433938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433845    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.433938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433927    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token\") pod \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\" (UID: \"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba\") "
Oct 30 21:36:49.434266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.433937    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:49.434266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434163    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:49.434266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434188    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:49.434552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434530    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:49.434627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434551    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.434627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434573    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.434627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434588    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.434795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.434636    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.437525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.437482    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "kube-api-access-ckxkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:49.437859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.437806    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:49.437968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.437836    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:36:49.439693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.439235    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token" pod "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" (UID: "5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:49.535484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.535446    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.535484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.535478    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.535484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.535487    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.535757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.535497    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.535757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.535507    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxkd\" (UniqueName: \"kubernetes.io/projected/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba-kube-api-access-ckxkd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:36:49.569565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.569475    6695 generic.go:356] "Generic (PLEG): container finished" podID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerID="ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5" exitCode=0
Oct 30 21:36:49.569565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.569531    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"
Oct 30 21:36:49.570082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.569564    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" event={"ID":"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba","Type":"ContainerDied","Data":"ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5"}
Oct 30 21:36:49.570082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.569637    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58" event={"ID":"5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba","Type":"ContainerDied","Data":"a85bec02ef3fd7b8b9fa22427e8237cf8c1fc51fcd77901a76d4d0104c92d0d3"}
Oct 30 21:36:49.570082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.569662    6695 scope.go:122] "RemoveContainer" containerID="ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5"
Oct 30 21:36:49.582806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.582786    6695 scope.go:122] "RemoveContainer" containerID="73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"
Oct 30 21:36:49.600999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.600979    6695 scope.go:122] "RemoveContainer" containerID="ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5"
Oct 30 21:36:49.601306 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:49.601280    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5\": container with ID starting with ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5 not found: ID does not exist" containerID="ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5"
Oct 30 21:36:49.601410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.601316    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5"} err="failed to get container status \"ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5\": rpc error: code = NotFound desc = could not find container \"ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5\": container with ID starting with ae43c5d05052e1fb0d23eac17596e9a93fba078ca37048e09148d6791ecb00f5 not found: ID does not exist"
Oct 30 21:36:49.601410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.601338    6695 scope.go:122] "RemoveContainer" containerID="73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"
Oct 30 21:36:49.601702 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:36:49.601677    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2\": container with ID starting with 73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2 not found: ID does not exist" containerID="73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"
Oct 30 21:36:49.601759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.601711    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2"} err="failed to get container status \"73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2\": rpc error: code = NotFound desc = could not find container \"73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2\": container with ID starting with 73a96b490f9b9c8020933022cc4111ad0651b7ce978e852378a01fc51848efd2 not found: ID does not exist"
Oct 30 21:36:49.627097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.627068    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"]
Oct 30 21:36:49.635268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.635237    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-38e5bf-openshift-default-6c69fffb75-ghb58"]
Oct 30 21:36:49.711500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:49.711472    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:50.578619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:50.578583    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:50.579544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:50.579521    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:36:51.215790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:51.215760    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" path="/var/lib/kubelet/pods/5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba/volumes"
Oct 30 21:36:53.161573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.161531    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"]
Oct 30 21:36:53.162301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.162279    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy"
Oct 30 21:36:53.162301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.162300    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy"
Oct 30 21:36:53.162451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.162332    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy"
Oct 30 21:36:53.162451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.162338    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy"
Oct 30 21:36:53.162569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.162451    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy"
Oct 30 21:36:53.165778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.165757    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.186548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.186512    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"]
Oct 30 21:36:53.196872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.194773    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-912e61\""
Oct 30 21:36:53.198853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.198816    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-912e61-dockercfg-lwx4l\""
Oct 30 21:36:53.200780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.200752    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"]
Oct 30 21:36:53.201998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.201976    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7dd11d-17cb-4ad5-b7ff-4341cfd9d0ba" containerName="istio-proxy"
Oct 30 21:36:53.205321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.205300    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.242676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.242404    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"]
Oct 30 21:36:53.327120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.283423    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.327739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.327700    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmhf\" (UniqueName: \"kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf\") pod \"payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.328015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.327989    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrxt\" (UniqueName: \"kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt\") pod \"payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.328195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.328180    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.429896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.429806    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.429896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.429882    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmhf\" (UniqueName: \"kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf\") pod \"payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.430130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.429924    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrxt\" (UniqueName: \"kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt\") pod \"payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.430130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.429963    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.430540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.430516    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.430705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.430687    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.448783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.448757    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrxt\" (UniqueName: \"kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt\") pod \"payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.448927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.448865    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmhf\" (UniqueName: \"kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf\") pod \"payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.479806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.479767    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:53.603644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.601887    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:53.787581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.787542    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"]
Oct 30 21:36:53.790804 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:53.790775    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fa6b5e_cad0_43e0_a034_12061c05b741.slice/crio-9427a82325b9a68312d12e99ef68d2bc72362270aaaca5734bea4cb15056515e WatchSource:0}: Error finding container 9427a82325b9a68312d12e99ef68d2bc72362270aaaca5734bea4cb15056515e: Status 404 returned error can't find the container with id 9427a82325b9a68312d12e99ef68d2bc72362270aaaca5734bea4cb15056515e
Oct 30 21:36:53.925616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:53.925561    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"]
Oct 30 21:36:53.930737 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:36:53.930706    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264a8f62_181f_4cdf_b5d1_359cb9c52925.slice/crio-2cf6fce02b24899a5dae011b1f8dc1889d48b6ad0663851edd37d8fdfed4168f WatchSource:0}: Error finding container 2cf6fce02b24899a5dae011b1f8dc1889d48b6ad0663851edd37d8fdfed4168f: Status 404 returned error can't find the container with id 2cf6fce02b24899a5dae011b1f8dc1889d48b6ad0663851edd37d8fdfed4168f
Oct 30 21:36:54.615042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.615002    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd" event={"ID":"b1fa6b5e-cad0-43e0-a034-12061c05b741","Type":"ContainerStarted","Data":"32d80a88c291c0ea790335999ebc25f953fcdcd39ca28307b8a0893f0168b766"}
Oct 30 21:36:54.615524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.615070    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd" event={"ID":"b1fa6b5e-cad0-43e0-a034-12061c05b741","Type":"ContainerStarted","Data":"9427a82325b9a68312d12e99ef68d2bc72362270aaaca5734bea4cb15056515e"}
Oct 30 21:36:54.615524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.615254    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:36:54.617804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.617775    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l" event={"ID":"264a8f62-181f-4cdf-b5d1-359cb9c52925","Type":"ContainerStarted","Data":"9fd69e4de920ca44810fc9ba0175210aa009fc4f8f28457665f78d13d284cb1a"}
Oct 30 21:36:54.617939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.617823    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l" event={"ID":"264a8f62-181f-4cdf-b5d1-359cb9c52925","Type":"ContainerStarted","Data":"2cf6fce02b24899a5dae011b1f8dc1889d48b6ad0663851edd37d8fdfed4168f"}
Oct 30 21:36:54.617939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.617848    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:36:54.653904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.653836    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd" podStartSLOduration=1.653817646 podStartE2EDuration="1.653817646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:54.652677891 +0000 UTC m=+1597.575136188" watchObservedRunningTime="2026-10-30 21:36:54.653817646 +0000 UTC m=+1597.576275945"
Oct 30 21:36:54.696869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:36:54.696795    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l" podStartSLOduration=1.6967743610000001 podStartE2EDuration="1.696774361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:54.694328055 +0000 UTC m=+1597.616786353" watchObservedRunningTime="2026-10-30 21:36:54.696774361 +0000 UTC m=+1597.619232659"
Oct 30 21:37:01.003163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:01.003130    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd_b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d/payload-processing/0.log"
Oct 30 21:37:01.080737 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:37:01.080698    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:37:01.756000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:01.755968    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs_fccddbf8-0957-46b5-b911-d9f80264887a/payload-processing/0.log"
Oct 30 21:37:01.973772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:01.973744    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs_fccddbf8-0957-46b5-b911-d9f80264887a/payload-processing/0.log"
Oct 30 21:37:02.604356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.604320    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"]
Oct 30 21:37:02.610786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.610756    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.626032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.625997    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"]
Oct 30 21:37:02.633480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.633453    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:02.633676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.633554    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:02.633844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.633808    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:02.633922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.633852    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:37:02.705769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.705384    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm4g\" (UniqueName: \"kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.705769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.705580    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.705769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.705687    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.807115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.807078    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.807325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.807141    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.807325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.807262    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqm4g\" (UniqueName: \"kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.808217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.808193    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.809789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.809759    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.828424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.828387    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqm4g\" (UniqueName: \"kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:02.929520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:02.929431    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:03.242055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:03.241987    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"]
Oct 30 21:37:03.246553 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:03.246520    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63cb0f2_01da_4ec8_b58c_5298b8c81eb3.slice/crio-6801e5fbf784ed53209f9fca280abac4d549be6bb7f9e4e4b423ad968322ad3d WatchSource:0}: Error finding container 6801e5fbf784ed53209f9fca280abac4d549be6bb7f9e4e4b423ad968322ad3d: Status 404 returned error can't find the container with id 6801e5fbf784ed53209f9fca280abac4d549be6bb7f9e4e4b423ad968322ad3d
Oct 30 21:37:03.697364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:03.697321    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p" event={"ID":"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3","Type":"ContainerStarted","Data":"5893cd34e8117d3241d91ec04618fbcb1ddfbeb7acc973c6c58ea0458fc3fa81"}
Oct 30 21:37:03.697922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:03.697435    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p" event={"ID":"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3","Type":"ContainerStarted","Data":"6801e5fbf784ed53209f9fca280abac4d549be6bb7f9e4e4b423ad968322ad3d"}
Oct 30 21:37:03.739953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:03.739849    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p" podStartSLOduration=1.739836109 podStartE2EDuration="1.739836109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:03.737315476 +0000 UTC m=+1606.659773774" watchObservedRunningTime="2026-10-30 21:37:03.739836109 +0000 UTC m=+1606.662294407"
Oct 30 21:37:04.706583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:04.706545    6695 generic.go:356] "Generic (PLEG): container finished" podID="d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" containerID="5893cd34e8117d3241d91ec04618fbcb1ddfbeb7acc973c6c58ea0458fc3fa81" exitCode=0
Oct 30 21:37:04.707005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:04.706655    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p" event={"ID":"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3","Type":"ContainerDied","Data":"5893cd34e8117d3241d91ec04618fbcb1ddfbeb7acc973c6c58ea0458fc3fa81"}
Oct 30 21:37:05.628842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.628807    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:37:05.631004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.628886    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:37:05.723368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.723336    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": read tcp 10.134.0.2:58180->10.134.0.189:15021: read: connection reset by peer" start-of-body=
Oct 30 21:37:05.724017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.723983    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": read tcp 10.134.0.2:58180->10.134.0.189:15021: read: connection reset by peer"
Oct 30 21:37:05.900158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.900087    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:05.984845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.984810    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca\") pod \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") "
Oct 30 21:37:05.985087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.985070    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g\" (UniqueName: \"kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g\") pod \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") "
Oct 30 21:37:05.985181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.985115    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca\") pod \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\" (UID: \"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3\") "
Oct 30 21:37:05.985554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.985526    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca" pod "d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" (UID: "d63cb0f2-01da-4ec8-b58c-5298b8c81eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:05.987490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.987462    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca" pod "d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" (UID: "d63cb0f2-01da-4ec8-b58c-5298b8c81eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:05.987665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:05.987646    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g" pod "d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" (UID: "d63cb0f2-01da-4ec8-b58c-5298b8c81eb3"). InnerVolumeSpecName "kube-api-access-hqm4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:06.086725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.086689    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqm4g\" (UniqueName: \"kubernetes.io/projected/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-kube-api-access-hqm4g\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:06.086725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.086723    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:06.086952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.086735    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:06.725346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.725312    6695 generic.go:356] "Generic (PLEG): container finished" podID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerID="16d05cc93d83599bd5c8430a943dc4423ab0a7f682e609c55428ce0131f82d0d" exitCode=0
Oct 30 21:37:06.725880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.725382    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" event={"ID":"221369ca-f9e3-4d12-8279-13e85182e7c9","Type":"ContainerDied","Data":"16d05cc93d83599bd5c8430a943dc4423ab0a7f682e609c55428ce0131f82d0d"}
Oct 30 21:37:06.725880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.725848    6695 scope.go:122] "RemoveContainer" containerID="16d05cc93d83599bd5c8430a943dc4423ab0a7f682e609c55428ce0131f82d0d"
Oct 30 21:37:06.728850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.728816    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:06.729099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.729078    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:06.729248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.729231    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:06.730532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.730510    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p" event={"ID":"d63cb0f2-01da-4ec8-b58c-5298b8c81eb3","Type":"ContainerDied","Data":"6801e5fbf784ed53209f9fca280abac4d549be6bb7f9e4e4b423ad968322ad3d"}
Oct 30 21:37:06.730719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.730703    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6801e5fbf784ed53209f9fca280abac4d549be6bb7f9e4e4b423ad968322ad3d"
Oct 30 21:37:06.730820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:06.730548    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"
Oct 30 21:37:07.747301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:07.747261    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" event={"ID":"221369ca-f9e3-4d12-8279-13e85182e7c9","Type":"ContainerStarted","Data":"ad578183f56b5614e7eb2b86cb133105a68a14470bc2f95896bb24df82058cfb"}
Oct 30 21:37:07.797806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:07.797771    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:37:07.798051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:07.798030    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 21:37:07.798102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:07.798084    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 21:37:08.798677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:08.798641    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 21:37:08.799089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:08.798700    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 21:37:09.798414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:09.798376    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 21:37:09.798671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:09.798503    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 21:37:10.798322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:10.798286    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 21:37:10.798748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:10.798340    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 21:37:11.031627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.026890    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"]
Oct 30 21:37:11.031627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.027986    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" containerName="revoke-keys"
Oct 30 21:37:11.031627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.028008    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" containerName="revoke-keys"
Oct 30 21:37:11.031627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.028153    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" containerName="revoke-keys"
Oct 30 21:37:11.038419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.038390    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.041618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.041574    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"]
Oct 30 21:37:11.064641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.064514    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:11.064812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.064713    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:11.064812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.064731    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:37:11.064812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.064742    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:11.193890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.193849    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.194070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.193910    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.194070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.194055    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw85c\" (UniqueName: \"kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.295178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.295143    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.295372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.295219    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.295372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.295334    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw85c\" (UniqueName: \"kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.295928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.295902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.297779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.297753    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.332351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.332275    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw85c\" (UniqueName: \"kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c\") pod \"maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.356868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.356833    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:11.703532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.703446    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"]
Oct 30 21:37:11.706791 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:11.706760    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ad76b8_4745_4686_892f_8ee996ae272c.slice/crio-be6d9d280a7c943ce4d338be91acc0919b0454e5c217115dbf9056d593f2aa02 WatchSource:0}: Error finding container be6d9d280a7c943ce4d338be91acc0919b0454e5c217115dbf9056d593f2aa02: Status 404 returned error can't find the container with id be6d9d280a7c943ce4d338be91acc0919b0454e5c217115dbf9056d593f2aa02
Oct 30 21:37:11.789022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.788985    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn" event={"ID":"18ad76b8-4745-4686-892f-8ee996ae272c","Type":"ContainerStarted","Data":"be6d9d280a7c943ce4d338be91acc0919b0454e5c217115dbf9056d593f2aa02"}
Oct 30 21:37:11.798465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.798428    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 21:37:11.798869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:11.798481    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 21:37:12.797822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:12.797787    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 21:37:12.798020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:12.797836    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 21:37:12.798262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:12.798228    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn" event={"ID":"18ad76b8-4745-4686-892f-8ee996ae272c","Type":"ContainerStarted","Data":"2561293cfa707f243dee3f1b361de947178bc072e62e3fb06430c7bc44a66573"}
Oct 30 21:37:12.837894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:12.837850    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn" podStartSLOduration=2.837837712 podStartE2EDuration="2.837837712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:12.837197427 +0000 UTC m=+1615.759655727" watchObservedRunningTime="2026-10-30 21:37:12.837837712 +0000 UTC m=+1615.760296010"
Oct 30 21:37:13.802211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:13.802174    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:37:13.802745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:13.802722    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:37:13.803498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:13.803476    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:37:13.807526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:13.807500    6695 generic.go:356] "Generic (PLEG): container finished" podID="18ad76b8-4745-4686-892f-8ee996ae272c" containerID="2561293cfa707f243dee3f1b361de947178bc072e62e3fb06430c7bc44a66573" exitCode=0
Oct 30 21:37:13.807697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:13.807571    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn" event={"ID":"18ad76b8-4745-4686-892f-8ee996ae272c","Type":"ContainerDied","Data":"2561293cfa707f243dee3f1b361de947178bc072e62e3fb06430c7bc44a66573"}
Oct 30 21:37:14.987849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:14.987822    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:15.062625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.062572    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c\" (UniqueName: \"kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c\") pod \"18ad76b8-4745-4686-892f-8ee996ae272c\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") "
Oct 30 21:37:15.062829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.062779    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca\") pod \"18ad76b8-4745-4686-892f-8ee996ae272c\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") "
Oct 30 21:37:15.062932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.062905    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca\") pod \"18ad76b8-4745-4686-892f-8ee996ae272c\" (UID: \"18ad76b8-4745-4686-892f-8ee996ae272c\") "
Oct 30 21:37:15.063299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.063266    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca" pod "18ad76b8-4745-4686-892f-8ee996ae272c" (UID: "18ad76b8-4745-4686-892f-8ee996ae272c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:15.063597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.063543    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:15.065594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.065565    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca" pod "18ad76b8-4745-4686-892f-8ee996ae272c" (UID: "18ad76b8-4745-4686-892f-8ee996ae272c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:15.065698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.065643    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c" pod "18ad76b8-4745-4686-892f-8ee996ae272c" (UID: "18ad76b8-4745-4686-892f-8ee996ae272c"). InnerVolumeSpecName "kube-api-access-pw85c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:15.165143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.165106    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw85c\" (UniqueName: \"kubernetes.io/projected/18ad76b8-4745-4686-892f-8ee996ae272c-kube-api-access-pw85c\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:15.165143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.165139    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad76b8-4745-4686-892f-8ee996ae272c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:15.677017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.676984    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"]
Oct 30 21:37:15.677877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.677854    6695 cpu_manager.go:414] "containerMap: removing container" podUID="18ad76b8-4745-4686-892f-8ee996ae272c" containerName="revoke-keys"
Oct 30 21:37:15.677877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.677876    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ad76b8-4745-4686-892f-8ee996ae272c" containerName="revoke-keys"
Oct 30 21:37:15.678066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.678054    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ad76b8-4745-4686-892f-8ee996ae272c" containerName="revoke-keys"
Oct 30 21:37:15.681693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.681673    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:15.755625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.754678    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"]
Oct 30 21:37:15.791011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.790964    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8ll\" (UniqueName: \"kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:15.791212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.791064    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:15.791468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.791449    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:15.953675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.924442    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:15.987955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.987913    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:16.015119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.015079    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8ll\" (UniqueName: \"kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:16.015516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.011924    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:16.015700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:15.989284    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:16.020203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.020137    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"
Oct 30 21:37:16.021377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.021346    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn" event={"ID":"18ad76b8-4745-4686-892f-8ee996ae272c","Type":"ContainerDied","Data":"be6d9d280a7c943ce4d338be91acc0919b0454e5c217115dbf9056d593f2aa02"}
Oct 30 21:37:16.021637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.021621    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be6d9d280a7c943ce4d338be91acc0919b0454e5c217115dbf9056d593f2aa02"
Oct 30 21:37:16.036867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.036823    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8ll\" (UniqueName: \"kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll\") pod \"maas-api-revoke-sub-f94acbb7-hqhsm\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:16.056709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.056679    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:16.372632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:16.370478    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"]
Oct 30 21:37:16.374795 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:16.374755    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501b4020_c079_48c1_9396_8fb7815a7635.slice/crio-a6093485667c485a3e8784f46e084425506742a01bf15230214bfeee841ec124 WatchSource:0}: Error finding container a6093485667c485a3e8784f46e084425506742a01bf15230214bfeee841ec124: Status 404 returned error can't find the container with id a6093485667c485a3e8784f46e084425506742a01bf15230214bfeee841ec124
Oct 30 21:37:17.028573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:17.028533    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm" event={"ID":"501b4020-c079-48c1-9396-8fb7815a7635","Type":"ContainerStarted","Data":"2d6c8d00030c85aa3e9a2d70fc7d5cfc330bbd2339db8341776f9f644ef86af6"}
Oct 30 21:37:17.029073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:17.028652    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm" event={"ID":"501b4020-c079-48c1-9396-8fb7815a7635","Type":"ContainerStarted","Data":"a6093485667c485a3e8784f46e084425506742a01bf15230214bfeee841ec124"}
Oct 30 21:37:17.068560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:17.068487    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm" podStartSLOduration=2.068467309 podStartE2EDuration="2.068467309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:17.066039388 +0000 UTC m=+1619.988497685" watchObservedRunningTime="2026-10-30 21:37:17.068467309 +0000 UTC m=+1619.990925608"
Oct 30 21:37:18.058628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:18.056774    6695 generic.go:356] "Generic (PLEG): container finished" podID="501b4020-c079-48c1-9396-8fb7815a7635" containerID="2d6c8d00030c85aa3e9a2d70fc7d5cfc330bbd2339db8341776f9f644ef86af6" exitCode=0
Oct 30 21:37:18.058628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:18.056869    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm" event={"ID":"501b4020-c079-48c1-9396-8fb7815a7635","Type":"ContainerDied","Data":"2d6c8d00030c85aa3e9a2d70fc7d5cfc330bbd2339db8341776f9f644ef86af6"}
Oct 30 21:37:19.058648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.054359    6695 scope.go:122] "RemoveContainer" containerID="982712d3616417f522f98dd3fdfee019487341d9ce09616e969127768eb363cd"
Oct 30 21:37:19.073228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.073202    6695 scope.go:122] "RemoveContainer" containerID="a3119f329f01709916975ccd66eb9f48c06f82bac2807620a12e679edde975be"
Oct 30 21:37:19.082815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.082796    6695 scope.go:122] "RemoveContainer" containerID="b9a17974b956ee90c33fba4e95bb49d7ed1ba65ee6390488062748fd82d62e73"
Oct 30 21:37:19.093567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.093546    6695 scope.go:122] "RemoveContainer" containerID="0ae9a856f9c501d5cb60b856fbfb4893dfd54ce579e57a4ee350b993159a7ee0"
Oct 30 21:37:19.380218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.380193    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:19.573176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.573139    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca\") pod \"501b4020-c079-48c1-9396-8fb7815a7635\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") "
Oct 30 21:37:19.573362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.573313    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca\") pod \"501b4020-c079-48c1-9396-8fb7815a7635\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") "
Oct 30 21:37:19.573407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.573396    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll\" (UniqueName: \"kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll\") pod \"501b4020-c079-48c1-9396-8fb7815a7635\" (UID: \"501b4020-c079-48c1-9396-8fb7815a7635\") "
Oct 30 21:37:19.574705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.574677    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca" pod "501b4020-c079-48c1-9396-8fb7815a7635" (UID: "501b4020-c079-48c1-9396-8fb7815a7635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:19.576118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.576095    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca" pod "501b4020-c079-48c1-9396-8fb7815a7635" (UID: "501b4020-c079-48c1-9396-8fb7815a7635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:19.577787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.577762    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll" pod "501b4020-c079-48c1-9396-8fb7815a7635" (UID: "501b4020-c079-48c1-9396-8fb7815a7635"). InnerVolumeSpecName "kube-api-access-9n8ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:19.675485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.675445    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:19.675485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.675489    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n8ll\" (UniqueName: \"kubernetes.io/projected/501b4020-c079-48c1-9396-8fb7815a7635-kube-api-access-9n8ll\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:19.675727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:19.675506    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b4020-c079-48c1-9396-8fb7815a7635-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:20.079162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:20.079129    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"
Oct 30 21:37:20.079646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:20.079169    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm" event={"ID":"501b4020-c079-48c1-9396-8fb7815a7635","Type":"ContainerDied","Data":"a6093485667c485a3e8784f46e084425506742a01bf15230214bfeee841ec124"}
Oct 30 21:37:20.079723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:20.079674    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6093485667c485a3e8784f46e084425506742a01bf15230214bfeee841ec124"
Oct 30 21:37:21.477794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:21.477758    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"]
Oct 30 21:37:21.514630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:21.511096    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"]
Oct 30 21:37:23.817122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.817086    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"]
Oct 30 21:37:23.818224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.818196    6695 cpu_manager.go:414] "containerMap: removing container" podUID="501b4020-c079-48c1-9396-8fb7815a7635" containerName="revoke-keys"
Oct 30 21:37:23.818224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.818220    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501b4020-c079-48c1-9396-8fb7815a7635" containerName="revoke-keys"
Oct 30 21:37:23.818430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.818390    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="501b4020-c079-48c1-9396-8fb7815a7635" containerName="revoke-keys"
Oct 30 21:37:23.824197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.824174    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:23.838954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.838929    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:23.853849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.850597    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:23.853849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.850702    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:37:23.853849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.850703    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:23.855856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.855832    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"]
Oct 30 21:37:23.915303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.915224    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:23.915303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.915290    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxtc\" (UniqueName: \"kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:23.915552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:23.915523    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.017068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.017022    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.017273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.017128    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxtc\" (UniqueName: \"kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.017350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.017322    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.017975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.017953    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.019693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.019665    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.035439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.035403    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxtc\" (UniqueName: \"kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc\") pod \"maas-api-revoke-keys-e2e-praxis-a-13658f-79x54\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.144752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.144714    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:24.458088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:24.458051    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"]
Oct 30 21:37:24.462229 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:24.462191    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0551363_9b5d_4e57_965e_1eb78384ad53.slice/crio-fc0d612249ec77be34dfe6dc4d252ca513cc4a9c1b8c3eb73bb7a090e220d28b WatchSource:0}: Error finding container fc0d612249ec77be34dfe6dc4d252ca513cc4a9c1b8c3eb73bb7a090e220d28b: Status 404 returned error can't find the container with id fc0d612249ec77be34dfe6dc4d252ca513cc4a9c1b8c3eb73bb7a090e220d28b
Oct 30 21:37:25.128095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:25.128007    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54" event={"ID":"e0551363-9b5d-4e57-965e-1eb78384ad53","Type":"ContainerStarted","Data":"87444b7edda468ff757e8a8d816290879309d245729a9f8e9c2413536249c2a4"}
Oct 30 21:37:25.128095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:25.128060    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54" event={"ID":"e0551363-9b5d-4e57-965e-1eb78384ad53","Type":"ContainerStarted","Data":"fc0d612249ec77be34dfe6dc4d252ca513cc4a9c1b8c3eb73bb7a090e220d28b"}
Oct 30 21:37:25.164197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:25.164122    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"]
Oct 30 21:37:25.168397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:25.168362    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3a5c92-vt2jn"]
Oct 30 21:37:25.173467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:25.173401    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54" podStartSLOduration=2.173383528 podStartE2EDuration="2.173383528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:25.166632902 +0000 UTC m=+1628.089091203" watchObservedRunningTime="2026-10-30 21:37:25.173383528 +0000 UTC m=+1628.095841826"
Oct 30 21:37:25.217328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:25.217280    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ad76b8-4745-4686-892f-8ee996ae272c" path="/var/lib/kubelet/pods/18ad76b8-4745-4686-892f-8ee996ae272c/volumes"
Oct 30 21:37:26.140071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.140031    6695 generic.go:356] "Generic (PLEG): container finished" podID="e0551363-9b5d-4e57-965e-1eb78384ad53" containerID="87444b7edda468ff757e8a8d816290879309d245729a9f8e9c2413536249c2a4" exitCode=0
Oct 30 21:37:26.140542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.140143    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54" event={"ID":"e0551363-9b5d-4e57-965e-1eb78384ad53","Type":"ContainerDied","Data":"87444b7edda468ff757e8a8d816290879309d245729a9f8e9c2413536249c2a4"}
Oct 30 21:37:26.479156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.479050    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68" podUID="f0937ce8-ff47-49dc-b7ea-057d1ba37168" containerName="payload-processing" containerID="cri-o://781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb" gracePeriod=25
Oct 30 21:37:26.511997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.511952    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n" podUID="0cc4f993-e60a-4ecd-8bca-156ef48091a8" containerName="payload-pre-processing" containerID="cri-o://5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4" gracePeriod=25
Oct 30 21:37:26.875811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.874988    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:37:26.913944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.913744    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2\" (UniqueName: \"kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2\") pod \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") "
Oct 30 21:37:26.913944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.913816    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume\") pod \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\" (UID: \"f0937ce8-ff47-49dc-b7ea-057d1ba37168\") "
Oct 30 21:37:26.915139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.914737    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume" pod "f0937ce8-ff47-49dc-b7ea-057d1ba37168" (UID: "f0937ce8-ff47-49dc-b7ea-057d1ba37168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:26.923418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.923098    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:37:26.924093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:26.923724    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2" pod "f0937ce8-ff47-49dc-b7ea-057d1ba37168" (UID: "f0937ce8-ff47-49dc-b7ea-057d1ba37168"). InnerVolumeSpecName "kube-api-access-cc9r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:27.014903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.014815    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9\" (UniqueName: \"kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9\") pod \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") "
Oct 30 21:37:27.015103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.014948    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume\") pod \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\" (UID: \"0cc4f993-e60a-4ecd-8bca-156ef48091a8\") "
Oct 30 21:37:27.015645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.015518    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9r2\" (UniqueName: \"kubernetes.io/projected/f0937ce8-ff47-49dc-b7ea-057d1ba37168-kube-api-access-cc9r2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.015645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.015546    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0937ce8-ff47-49dc-b7ea-057d1ba37168-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.015645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.015585    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume" pod "0cc4f993-e60a-4ecd-8bca-156ef48091a8" (UID: "0cc4f993-e60a-4ecd-8bca-156ef48091a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:27.020105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.020075    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9" pod "0cc4f993-e60a-4ecd-8bca-156ef48091a8" (UID: "0cc4f993-e60a-4ecd-8bca-156ef48091a8"). InnerVolumeSpecName "kube-api-access-756g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:27.117332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.117294    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756g9\" (UniqueName: \"kubernetes.io/projected/0cc4f993-e60a-4ecd-8bca-156ef48091a8-kube-api-access-756g9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.117332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.117330    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc4f993-e60a-4ecd-8bca-156ef48091a8-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.150450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.150412    6695 generic.go:356] "Generic (PLEG): container finished" podID="0cc4f993-e60a-4ecd-8bca-156ef48091a8" containerID="5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4" exitCode=0
Oct 30 21:37:27.150986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.150460    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"
Oct 30 21:37:27.150986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.150492    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n" event={"ID":"0cc4f993-e60a-4ecd-8bca-156ef48091a8","Type":"ContainerDied","Data":"5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4"}
Oct 30 21:37:27.150986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.150628    6695 scope.go:122] "RemoveContainer" containerID="5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4"
Oct 30 21:37:27.150986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.150597    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n" event={"ID":"0cc4f993-e60a-4ecd-8bca-156ef48091a8","Type":"ContainerDied","Data":"3f48dcdc598e62d48f623b7082ff22ea1b38cdd933ef46f39e4fa88641b5a223"}
Oct 30 21:37:27.152783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.152754    6695 generic.go:356] "Generic (PLEG): container finished" podID="f0937ce8-ff47-49dc-b7ea-057d1ba37168" containerID="781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb" exitCode=0
Oct 30 21:37:27.152935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.152817    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"
Oct 30 21:37:27.152935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.152839    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68" event={"ID":"f0937ce8-ff47-49dc-b7ea-057d1ba37168","Type":"ContainerDied","Data":"781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb"}
Oct 30 21:37:27.152935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.152897    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68" event={"ID":"f0937ce8-ff47-49dc-b7ea-057d1ba37168","Type":"ContainerDied","Data":"38d742a49fb0ad31aa675b0328d1b133e5a58a0dda0cb21e8e97d5837c8f82fd"}
Oct 30 21:37:27.166169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.165767    6695 scope.go:122] "RemoveContainer" containerID="5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4"
Oct 30 21:37:27.166169 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:37:27.166048    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4\": container with ID starting with 5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4 not found: ID does not exist" containerID="5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4"
Oct 30 21:37:27.166169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.166086    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4"} err="failed to get container status \"5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4\": rpc error: code = NotFound desc = could not find container \"5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4\": container with ID starting with 5aafdb58c793304fe77df6ce6a5e390b8b29e99c2a68aa2cfceb80b21695adf4 not found: ID does not exist"
Oct 30 21:37:27.166169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.166109    6695 scope.go:122] "RemoveContainer" containerID="781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb"
Oct 30 21:37:27.179472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.179333    6695 scope.go:122] "RemoveContainer" containerID="781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb"
Oct 30 21:37:27.179960 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:37:27.179916    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb\": container with ID starting with 781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb not found: ID does not exist" containerID="781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb"
Oct 30 21:37:27.179960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.179942    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb"} err="failed to get container status \"781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb\": rpc error: code = NotFound desc = could not find container \"781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb\": container with ID starting with 781ed4f10c950c23961c2ad217af1d8f0a1340b7dc73f11416f7eb9796d07ebb not found: ID does not exist"
Oct 30 21:37:27.204519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.204488    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"]
Oct 30 21:37:27.218723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.218690    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3a5c92-595945f747-tp54n"]
Oct 30 21:37:27.287697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.287665    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"]
Oct 30 21:37:27.290697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.290669    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3a5c92-69b4f978cc-qrf68"]
Oct 30 21:37:27.378689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.378659    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:27.422697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.422667    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc\" (UniqueName: \"kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc\") pod \"e0551363-9b5d-4e57-965e-1eb78384ad53\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") "
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.422802    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca\") pod \"e0551363-9b5d-4e57-965e-1eb78384ad53\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") "
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.422866    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca\") pod \"e0551363-9b5d-4e57-965e-1eb78384ad53\" (UID: \"e0551363-9b5d-4e57-965e-1eb78384ad53\") "
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.423207    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca" pod "e0551363-9b5d-4e57-965e-1eb78384ad53" (UID: "e0551363-9b5d-4e57-965e-1eb78384ad53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.423383    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.424930    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc" pod "e0551363-9b5d-4e57-965e-1eb78384ad53" (UID: "e0551363-9b5d-4e57-965e-1eb78384ad53"). InnerVolumeSpecName "kube-api-access-pzxtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.424927    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca" pod "e0551363-9b5d-4e57-965e-1eb78384ad53" (UID: "e0551363-9b5d-4e57-965e-1eb78384ad53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.524217    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxtc\" (UniqueName: \"kubernetes.io/projected/e0551363-9b5d-4e57-965e-1eb78384ad53-kube-api-access-pzxtc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:27.699200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:27.524238    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0551363-9b5d-4e57-965e-1eb78384ad53-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:28.163016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.162989    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"
Oct 30 21:37:28.180817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.163027    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54" event={"ID":"e0551363-9b5d-4e57-965e-1eb78384ad53","Type":"ContainerDied","Data":"fc0d612249ec77be34dfe6dc4d252ca513cc4a9c1b8c3eb73bb7a090e220d28b"}
Oct 30 21:37:28.180817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.163088    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc0d612249ec77be34dfe6dc4d252ca513cc4a9c1b8c3eb73bb7a090e220d28b"
Oct 30 21:37:28.351420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.351379    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"]
Oct 30 21:37:28.352318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352298    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0cc4f993-e60a-4ecd-8bca-156ef48091a8" containerName="payload-pre-processing"
Oct 30 21:37:28.352389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352321    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc4f993-e60a-4ecd-8bca-156ef48091a8" containerName="payload-pre-processing"
Oct 30 21:37:28.352389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352365    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e0551363-9b5d-4e57-965e-1eb78384ad53" containerName="revoke-keys"
Oct 30 21:37:28.352389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352375    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0551363-9b5d-4e57-965e-1eb78384ad53" containerName="revoke-keys"
Oct 30 21:37:28.352496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352389    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f0937ce8-ff47-49dc-b7ea-057d1ba37168" containerName="payload-processing"
Oct 30 21:37:28.352496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352397    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0937ce8-ff47-49dc-b7ea-057d1ba37168" containerName="payload-processing"
Oct 30 21:37:28.352583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352572    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc4f993-e60a-4ecd-8bca-156ef48091a8" containerName="payload-pre-processing"
Oct 30 21:37:28.352641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352591    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0551363-9b5d-4e57-965e-1eb78384ad53" containerName="revoke-keys"
Oct 30 21:37:28.352641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.352621    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0937ce8-ff47-49dc-b7ea-057d1ba37168" containerName="payload-processing"
Oct 30 21:37:28.375598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.375567    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"]
Oct 30 21:37:28.375801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.375726    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.389311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.389284    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:28.389754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.389636    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:37:28.389878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.389804    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:28.390388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.389938    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:28.435857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.435779    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.436018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.435894    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.436018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.435921    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44t5\" (UniqueName: \"kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.537163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.537135    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.537340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.537198    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.537340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.537228    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44t5\" (UniqueName: \"kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.537888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.537869    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.539374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.539347    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.556555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.556532    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44t5\" (UniqueName: \"kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5\") pod \"maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:28.710187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:28.710114    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:29.015342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.015229    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"]
Oct 30 21:37:29.018635 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:29.018580    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a5b4f5_6598_4d39_bc8a_6fb29b8eae08.slice/crio-94e1ba2ae752857f6873b6768187ab580ed5d17e6bc85a0668bc238b5d73b5a2 WatchSource:0}: Error finding container 94e1ba2ae752857f6873b6768187ab580ed5d17e6bc85a0668bc238b5d73b5a2: Status 404 returned error can't find the container with id 94e1ba2ae752857f6873b6768187ab580ed5d17e6bc85a0668bc238b5d73b5a2
Oct 30 21:37:29.172198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.172169    6695 generic.go:356] "Generic (PLEG): container finished" podID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerID="2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2" exitCode=0
Oct 30 21:37:29.172689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.172250    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" event={"ID":"c482d66f-1a6c-4407-8c32-20b4fb649d28","Type":"ContainerDied","Data":"2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"}
Oct 30 21:37:29.172825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.172807    6695 scope.go:122] "RemoveContainer" containerID="2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"
Oct 30 21:37:29.174083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.174055    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n" event={"ID":"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08","Type":"ContainerStarted","Data":"94e1ba2ae752857f6873b6768187ab580ed5d17e6bc85a0668bc238b5d73b5a2"}
Oct 30 21:37:29.174498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.174475    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:29.174582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.174549    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:29.174649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.174596    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:29.176537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.176512    6695 generic.go:356] "Generic (PLEG): container finished" podID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerID="f615c346ad4f1deba3c7580da28aba46133d6e7af342e49a070444bd36274b5c" exitCode=0
Oct 30 21:37:29.176679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.176629    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" event={"ID":"f7c1f544-3c15-41fe-aed1-17449c0e7766","Type":"ContainerDied","Data":"f615c346ad4f1deba3c7580da28aba46133d6e7af342e49a070444bd36274b5c"}
Oct 30 21:37:29.177057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.177040    6695 scope.go:122] "RemoveContainer" containerID="f615c346ad4f1deba3c7580da28aba46133d6e7af342e49a070444bd36274b5c"
Oct 30 21:37:29.178808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.178777    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:29.178890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.178866    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:29.178939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.178922    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:29.215072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.215042    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc4f993-e60a-4ecd-8bca-156ef48091a8" path="/var/lib/kubelet/pods/0cc4f993-e60a-4ecd-8bca-156ef48091a8/volumes"
Oct 30 21:37:29.215526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:29.215510    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0937ce8-ff47-49dc-b7ea-057d1ba37168" path="/var/lib/kubelet/pods/f0937ce8-ff47-49dc-b7ea-057d1ba37168/volumes"
Oct 30 21:37:30.189308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.189265    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" event={"ID":"c482d66f-1a6c-4407-8c32-20b4fb649d28","Type":"ContainerStarted","Data":"31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5"}
Oct 30 21:37:30.192270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.192243    6695 generic.go:356] "Generic (PLEG): container finished" podID="15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" containerID="816dd2bbe875066263c1d1e58e2baa3f349dd99e76913e0f8382f16faeea5977" exitCode=0
Oct 30 21:37:30.192438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.192407    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n" event={"ID":"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08","Type":"ContainerDied","Data":"816dd2bbe875066263c1d1e58e2baa3f349dd99e76913e0f8382f16faeea5977"}
Oct 30 21:37:30.194613 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.194574    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" event={"ID":"f7c1f544-3c15-41fe-aed1-17449c0e7766","Type":"ContainerStarted","Data":"ca2546b0a76f691ffcdf89a9237b374b1d891bd6b9d012fd555a69a46df0474b"}
Oct 30 21:37:30.303258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.303227    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:37:30.303568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.303543    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 21:37:30.303649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.303627    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 21:37:30.907523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.907484    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:37:30.909133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.909098    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 21:37:30.909312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:30.909160    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 21:37:31.249102 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:37:31.249057    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:37:31.306995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.306962    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 21:37:31.307280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.307237    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 21:37:31.797154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.797121    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:31.872652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.870800    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca\") pod \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") "
Oct 30 21:37:31.872652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.870855    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5\" (UniqueName: \"kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5\") pod \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") "
Oct 30 21:37:31.872652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.870964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca\") pod \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\" (UID: \"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08\") "
Oct 30 21:37:31.875970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.873701    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca" pod "15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" (UID: "15a5b4f5-6598-4d39-bc8a-6fb29b8eae08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:31.882467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.882241    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5" pod "15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" (UID: "15a5b4f5-6598-4d39-bc8a-6fb29b8eae08"). InnerVolumeSpecName "kube-api-access-x44t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:31.882467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.882331    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca" pod "15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" (UID: "15a5b4f5-6598-4d39-bc8a-6fb29b8eae08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:31.908298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.908148    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 21:37:31.908298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.908221    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 21:37:31.975535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.975477    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:31.975535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.975518    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x44t5\" (UniqueName: \"kubernetes.io/projected/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-kube-api-access-x44t5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:31.975535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:31.975535    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.276832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.276794    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"
Oct 30 21:37:32.277561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.276811    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n" event={"ID":"15a5b4f5-6598-4d39-bc8a-6fb29b8eae08","Type":"ContainerDied","Data":"94e1ba2ae752857f6873b6768187ab580ed5d17e6bc85a0668bc238b5d73b5a2"}
Oct 30 21:37:32.277800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.277781    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e1ba2ae752857f6873b6768187ab580ed5d17e6bc85a0668bc238b5d73b5a2"
Oct 30 21:37:32.306677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.306213    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 21:37:32.306677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.306273    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 21:37:32.923630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.910937    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 21:37:32.923630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:32.910998    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 21:37:33.302694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.302653    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 21:37:33.303159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.302715    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 21:37:33.306868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.306839    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"]
Oct 30 21:37:33.307499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.307485    6695 cpu_manager.go:414] "containerMap: removing container" podUID="15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" containerName="revoke-keys"
Oct 30 21:37:33.307557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.307500    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" containerName="revoke-keys"
Oct 30 21:37:33.307632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.307620    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" containerName="revoke-keys"
Oct 30 21:37:33.310057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.310037    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.362629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.342536    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:33.362629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.347110    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:33.362629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.348224    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"]
Oct 30 21:37:33.362629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.356078    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:37:33.362629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.356145    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:33.435627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.435445    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.435627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.435629    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.435907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.435680    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdc7m\" (UniqueName: \"kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.547674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.537533    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.547674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.537719    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.547674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.537770    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdc7m\" (UniqueName: \"kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.547674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.538976    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.547674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.543432    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.566164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.566087    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdc7m\" (UniqueName: \"kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m\") pod \"maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.697140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.697101    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:33.908348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.907540    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 21:37:33.908348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:33.907587    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 21:37:34.067168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.067125    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"]
Oct 30 21:37:34.068527 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:34.068505    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d996ad_2f54_462d_9b9c_026f7fb5d870.slice/crio-af7b6a3e528a6ce31735a2b73251c4811229f37bc24e0e7ecd57ae46f04610f6 WatchSource:0}: Error finding container af7b6a3e528a6ce31735a2b73251c4811229f37bc24e0e7ecd57ae46f04610f6: Status 404 returned error can't find the container with id af7b6a3e528a6ce31735a2b73251c4811229f37bc24e0e7ecd57ae46f04610f6
Oct 30 21:37:34.294241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.294204    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx" event={"ID":"a5d996ad-2f54-462d-9b9c-026f7fb5d870","Type":"ContainerStarted","Data":"af7b6a3e528a6ce31735a2b73251c4811229f37bc24e0e7ecd57ae46f04610f6"}
Oct 30 21:37:34.302539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.302509    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 21:37:34.302715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.302564    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 21:37:34.908522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.908436    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 21:37:34.908522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.908490    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 21:37:34.939822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:34.939789    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:37:35.302913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:35.302878    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 30 21:37:35.303331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:35.302939    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 30 21:37:35.304964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:35.304933    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx" event={"ID":"a5d996ad-2f54-462d-9b9c-026f7fb5d870","Type":"ContainerStarted","Data":"550c8ea09103ec98ce9b72edcc342401b71684faff8ba175c901a8851bee9f46"}
Oct 30 21:37:35.357919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:35.354080    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx" podStartSLOduration=2.354057707 podStartE2EDuration="2.354057707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:35.352030266 +0000 UTC m=+1638.274488564" watchObservedRunningTime="2026-10-30 21:37:35.354057707 +0000 UTC m=+1638.276516005"
Oct 30 21:37:35.907565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:35.907524    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 21:37:35.907751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:35.907591    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 21:37:36.319521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.319487    6695 generic.go:356] "Generic (PLEG): container finished" podID="a5d996ad-2f54-462d-9b9c-026f7fb5d870" containerID="550c8ea09103ec98ce9b72edcc342401b71684faff8ba175c901a8851bee9f46" exitCode=0
Oct 30 21:37:36.320060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.319596    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx" event={"ID":"a5d996ad-2f54-462d-9b9c-026f7fb5d870","Type":"ContainerDied","Data":"550c8ea09103ec98ce9b72edcc342401b71684faff8ba175c901a8851bee9f46"}
Oct 30 21:37:36.371291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.371256    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:37:36.372390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.372355    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:37:36.375908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.373488    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:37:36.925018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.924982    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:37:36.930274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:36.930248    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:37:37.490251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.490221    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:37.523729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.523694    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m\" (UniqueName: \"kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m\") pod \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") "
Oct 30 21:37:37.523912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.523760    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca\") pod \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") "
Oct 30 21:37:37.523912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.523901    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca\") pod \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\" (UID: \"a5d996ad-2f54-462d-9b9c-026f7fb5d870\") "
Oct 30 21:37:37.524631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.524550    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca" pod "a5d996ad-2f54-462d-9b9c-026f7fb5d870" (UID: "a5d996ad-2f54-462d-9b9c-026f7fb5d870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:37.524879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.524846    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:37.526160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.526130    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m" pod "a5d996ad-2f54-462d-9b9c-026f7fb5d870" (UID: "a5d996ad-2f54-462d-9b9c-026f7fb5d870"). InnerVolumeSpecName "kube-api-access-mdc7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:37.526584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.526557    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca" pod "a5d996ad-2f54-462d-9b9c-026f7fb5d870" (UID: "a5d996ad-2f54-462d-9b9c-026f7fb5d870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:37.625738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.625650    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdc7m\" (UniqueName: \"kubernetes.io/projected/a5d996ad-2f54-462d-9b9c-026f7fb5d870-kube-api-access-mdc7m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:37.625738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:37.625683    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5d996ad-2f54-462d-9b9c-026f7fb5d870-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.338971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:38.338936    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"
Oct 30 21:37:38.338971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:38.338966    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx" event={"ID":"a5d996ad-2f54-462d-9b9c-026f7fb5d870","Type":"ContainerDied","Data":"af7b6a3e528a6ce31735a2b73251c4811229f37bc24e0e7ecd57ae46f04610f6"}
Oct 30 21:37:38.339191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:38.339016    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7b6a3e528a6ce31735a2b73251c4811229f37bc24e0e7ecd57ae46f04610f6"
Oct 30 21:37:38.847379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:38.847336    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"]
Oct 30 21:37:42.218767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.218730    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"]
Oct 30 21:37:42.219148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.219031    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd" podUID="b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" containerName="payload-processing" containerID="cri-o://204e10341fe678156365ff235360afd28b0f952ff13bce28e82ca2fc3f0dce08" gracePeriod=30
Oct 30 21:37:42.235924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.235894    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"]
Oct 30 21:37:42.236151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.236127    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh" podUID="8b11200d-d525-4395-8fbc-220b70efaeea" containerName="payload-pre-processing" containerID="cri-o://3198703d64c987a1005ae48a3cfd031d0f1b92bd1454b27caee8b4f0493c5117" gracePeriod=30
Oct 30 21:37:42.371417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.371380    6695 generic.go:356] "Generic (PLEG): container finished" podID="8b11200d-d525-4395-8fbc-220b70efaeea" containerID="3198703d64c987a1005ae48a3cfd031d0f1b92bd1454b27caee8b4f0493c5117" exitCode=0
Oct 30 21:37:42.371598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.371468    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh" event={"ID":"8b11200d-d525-4395-8fbc-220b70efaeea","Type":"ContainerDied","Data":"3198703d64c987a1005ae48a3cfd031d0f1b92bd1454b27caee8b4f0493c5117"}
Oct 30 21:37:42.373461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.373432    6695 generic.go:356] "Generic (PLEG): container finished" podID="b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" containerID="204e10341fe678156365ff235360afd28b0f952ff13bce28e82ca2fc3f0dce08" exitCode=0
Oct 30 21:37:42.373641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.373482    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd" event={"ID":"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d","Type":"ContainerDied","Data":"204e10341fe678156365ff235360afd28b0f952ff13bce28e82ca2fc3f0dce08"}
Oct 30 21:37:42.642398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.642084    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:37:42.746649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.744270    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb\" (UniqueName: \"kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb\") pod \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") "
Oct 30 21:37:42.746649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.744454    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume\") pod \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\" (UID: \"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d\") "
Oct 30 21:37:42.746649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.745234    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume" pod "b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" (UID: "b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:42.747001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.746975    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:42.764042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.764013    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb" pod "b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" (UID: "b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d"). InnerVolumeSpecName "kube-api-access-xmgsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:42.765461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.765439    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:37:42.847403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.847366    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume\") pod \"8b11200d-d525-4395-8fbc-220b70efaeea\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") "
Oct 30 21:37:42.847692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.847667    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg\" (UniqueName: \"kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg\") pod \"8b11200d-d525-4395-8fbc-220b70efaeea\" (UID: \"8b11200d-d525-4395-8fbc-220b70efaeea\") "
Oct 30 21:37:42.847823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.847796    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume" pod "8b11200d-d525-4395-8fbc-220b70efaeea" (UID: "8b11200d-d525-4395-8fbc-220b70efaeea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:42.848290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.848263    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11200d-d525-4395-8fbc-220b70efaeea-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:42.848376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.848295    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmgsb\" (UniqueName: \"kubernetes.io/projected/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d-kube-api-access-xmgsb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:42.849977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.849940    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg" pod "8b11200d-d525-4395-8fbc-220b70efaeea" (UID: "8b11200d-d525-4395-8fbc-220b70efaeea"). InnerVolumeSpecName "kube-api-access-v2jgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:42.876195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.876131    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"]
Oct 30 21:37:42.879494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.879470    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1ee3-7lh9p"]
Oct 30 21:37:42.949710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:42.949678    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jgg\" (UniqueName: \"kubernetes.io/projected/8b11200d-d525-4395-8fbc-220b70efaeea-kube-api-access-v2jgg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:43.214747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.214713    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63cb0f2-01da-4ec8-b58c-5298b8c81eb3" path="/var/lib/kubelet/pods/d63cb0f2-01da-4ec8-b58c-5298b8c81eb3/volumes"
Oct 30 21:37:43.382535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.382509    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"
Oct 30 21:37:43.383021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.382551    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh" event={"ID":"8b11200d-d525-4395-8fbc-220b70efaeea","Type":"ContainerDied","Data":"d8ac706e9cd9ef195efd0704e5fb8c88130f5b3ccf88aca90c13b44076ecaef4"}
Oct 30 21:37:43.383021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.382624    6695 scope.go:122] "RemoveContainer" containerID="3198703d64c987a1005ae48a3cfd031d0f1b92bd1454b27caee8b4f0493c5117"
Oct 30 21:37:43.384099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.384075    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"
Oct 30 21:37:43.384183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.384120    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd" event={"ID":"b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d","Type":"ContainerDied","Data":"c69d04cf5a88a4e6426ad983deec4dc7d68bd831105f9f99628cba673bfcda11"}
Oct 30 21:37:43.393017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.392995    6695 scope.go:122] "RemoveContainer" containerID="204e10341fe678156365ff235360afd28b0f952ff13bce28e82ca2fc3f0dce08"
Oct 30 21:37:43.435051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.435014    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"]
Oct 30 21:37:43.445598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.445569    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a1ee3-69d46d7c48-wtfzd"]
Oct 30 21:37:43.476235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.476159    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"]
Oct 30 21:37:43.480235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.480198    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-7ff9c4c475-4fpgh"]
Oct 30 21:37:43.790451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.790416    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"]
Oct 30 21:37:43.823501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.823469    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"]
Oct 30 21:37:43.848630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:43.848579    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph" podUID="4a08fad2-c55e-4560-b7b5-0eddf63903a5" containerName="payload-pre-processing" containerID="cri-o://d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea" gracePeriod=25
Oct 30 21:37:44.190709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.190678    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:37:44.288250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.288214    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r\" (UniqueName: \"kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r\") pod \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") "
Oct 30 21:37:44.288421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.288399    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume\") pod \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\" (UID: \"4a08fad2-c55e-4560-b7b5-0eddf63903a5\") "
Oct 30 21:37:44.288789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.288759    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume" pod "4a08fad2-c55e-4560-b7b5-0eddf63903a5" (UID: "4a08fad2-c55e-4560-b7b5-0eddf63903a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:44.290252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.290231    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r" pod "4a08fad2-c55e-4560-b7b5-0eddf63903a5" (UID: "4a08fad2-c55e-4560-b7b5-0eddf63903a5"). InnerVolumeSpecName "kube-api-access-5gc4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:44.390877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.390841    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gc4r\" (UniqueName: \"kubernetes.io/projected/4a08fad2-c55e-4560-b7b5-0eddf63903a5-kube-api-access-5gc4r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:44.390877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.390880    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a08fad2-c55e-4560-b7b5-0eddf63903a5-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:44.395522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.395491    6695 generic.go:356] "Generic (PLEG): container finished" podID="4a08fad2-c55e-4560-b7b5-0eddf63903a5" containerID="d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea" exitCode=0
Oct 30 21:37:44.395737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.395555    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"
Oct 30 21:37:44.395737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.395578    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph" event={"ID":"4a08fad2-c55e-4560-b7b5-0eddf63903a5","Type":"ContainerDied","Data":"d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea"}
Oct 30 21:37:44.395737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.395682    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph" event={"ID":"4a08fad2-c55e-4560-b7b5-0eddf63903a5","Type":"ContainerDied","Data":"081eccebe5b269dbefb0241d6cd352252056932fc4a0d03ff2b7c35997054f10"}
Oct 30 21:37:44.395737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.395713    6695 scope.go:122] "RemoveContainer" containerID="d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea"
Oct 30 21:37:44.408474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.408457    6695 scope.go:122] "RemoveContainer" containerID="d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea"
Oct 30 21:37:44.408760 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:37:44.408739    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea\": container with ID starting with d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea not found: ID does not exist" containerID="d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea"
Oct 30 21:37:44.408840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.408772    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea"} err="failed to get container status \"d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea\": rpc error: code = NotFound desc = could not find container \"d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea\": container with ID starting with d5f809a1fe5acae631fd104bd126687c3dc40f7f2fb707db89042312ced282ea not found: ID does not exist"
Oct 30 21:37:44.480625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.479699    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"]
Oct 30 21:37:44.503635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:44.501302    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-26a13f-54c9d7c95-dvgph"]
Oct 30 21:37:45.225627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:45.224411    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a08fad2-c55e-4560-b7b5-0eddf63903a5" path="/var/lib/kubelet/pods/4a08fad2-c55e-4560-b7b5-0eddf63903a5/volumes"
Oct 30 21:37:45.225627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:45.225152    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b11200d-d525-4395-8fbc-220b70efaeea" path="/var/lib/kubelet/pods/8b11200d-d525-4395-8fbc-220b70efaeea/volumes"
Oct 30 21:37:45.225907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:45.225719    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" path="/var/lib/kubelet/pods/b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d/volumes"
Oct 30 21:37:48.339625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:48.338921    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"]
Oct 30 21:37:48.790711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:48.790673    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c" podUID="ab850567-4e74-48e9-9ff5-14795d4e0bc5" containerName="payload-processing" containerID="cri-o://056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989" gracePeriod=25
Oct 30 21:37:48.832636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:48.824904    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm" podUID="d26a5b26-e3bd-4250-974a-86400742b17b" containerName="payload-pre-processing" containerID="cri-o://993ca4b8e70a02b38d62e78f0dcfd27e273b011be876476e846573d9f47fc7b5" gracePeriod=25
Oct 30 21:37:49.424628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.423791    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:37:49.460186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.460158    6695 generic.go:356] "Generic (PLEG): container finished" podID="d26a5b26-e3bd-4250-974a-86400742b17b" containerID="993ca4b8e70a02b38d62e78f0dcfd27e273b011be876476e846573d9f47fc7b5" exitCode=0
Oct 30 21:37:49.460320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.460241    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm" event={"ID":"d26a5b26-e3bd-4250-974a-86400742b17b","Type":"ContainerDied","Data":"993ca4b8e70a02b38d62e78f0dcfd27e273b011be876476e846573d9f47fc7b5"}
Oct 30 21:37:49.460320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.460306    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm" event={"ID":"d26a5b26-e3bd-4250-974a-86400742b17b","Type":"ContainerDied","Data":"a6132fe2619fae342e849c0494a9eb0c2d31a776a8579806b992958eff40bbff"}
Oct 30 21:37:49.460426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.460331    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6132fe2619fae342e849c0494a9eb0c2d31a776a8579806b992958eff40bbff"
Oct 30 21:37:49.461632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.461596    6695 generic.go:356] "Generic (PLEG): container finished" podID="ab850567-4e74-48e9-9ff5-14795d4e0bc5" containerID="056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989" exitCode=0
Oct 30 21:37:49.461757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.461641    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c" event={"ID":"ab850567-4e74-48e9-9ff5-14795d4e0bc5","Type":"ContainerDied","Data":"056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989"}
Oct 30 21:37:49.461757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.461664    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"
Oct 30 21:37:49.461757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.461693    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c" event={"ID":"ab850567-4e74-48e9-9ff5-14795d4e0bc5","Type":"ContainerDied","Data":"82572cb83aa712b2f0729c483078e151fb3a14765834b0390a3cf1265dd13f5c"}
Oct 30 21:37:49.461757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.461723    6695 scope.go:122] "RemoveContainer" containerID="056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989"
Oct 30 21:37:49.468760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.468736    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:37:49.474725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.474699    6695 scope.go:122] "RemoveContainer" containerID="056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989"
Oct 30 21:37:49.475049 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:37:49.475022    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989\": container with ID starting with 056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989 not found: ID does not exist" containerID="056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989"
Oct 30 21:37:49.475149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.475080    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989"} err="failed to get container status \"056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989\": rpc error: code = NotFound desc = could not find container \"056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989\": container with ID starting with 056a688d48ea8fcf6c83106789d7b4f9fc350b5875a39f44e6a86f21348e7989 not found: ID does not exist"
Oct 30 21:37:49.583596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.583559    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n\" (UniqueName: \"kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n\") pod \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") "
Oct 30 21:37:49.583828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.583656    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume\") pod \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\" (UID: \"ab850567-4e74-48e9-9ff5-14795d4e0bc5\") "
Oct 30 21:37:49.583907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.583890    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume\") pod \"d26a5b26-e3bd-4250-974a-86400742b17b\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") "
Oct 30 21:37:49.583961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.583948    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42\" (UniqueName: \"kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42\") pod \"d26a5b26-e3bd-4250-974a-86400742b17b\" (UID: \"d26a5b26-e3bd-4250-974a-86400742b17b\") "
Oct 30 21:37:49.584072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.584049    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume" pod "ab850567-4e74-48e9-9ff5-14795d4e0bc5" (UID: "ab850567-4e74-48e9-9ff5-14795d4e0bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:49.584393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.584376    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume" pod "d26a5b26-e3bd-4250-974a-86400742b17b" (UID: "d26a5b26-e3bd-4250-974a-86400742b17b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:49.584937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.584913    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26a5b26-e3bd-4250-974a-86400742b17b-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:49.585020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.584941    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab850567-4e74-48e9-9ff5-14795d4e0bc5-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:49.586305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.586279    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n" pod "ab850567-4e74-48e9-9ff5-14795d4e0bc5" (UID: "ab850567-4e74-48e9-9ff5-14795d4e0bc5"). InnerVolumeSpecName "kube-api-access-mq85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:49.586397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.586349    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42" pod "d26a5b26-e3bd-4250-974a-86400742b17b" (UID: "d26a5b26-e3bd-4250-974a-86400742b17b"). InnerVolumeSpecName "kube-api-access-kmw42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:49.686677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.686568    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmw42\" (UniqueName: \"kubernetes.io/projected/d26a5b26-e3bd-4250-974a-86400742b17b-kube-api-access-kmw42\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:49.686677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.686637    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq85n\" (UniqueName: \"kubernetes.io/projected/ab850567-4e74-48e9-9ff5-14795d4e0bc5-kube-api-access-mq85n\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:37:49.806889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.806849    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"]
Oct 30 21:37:49.834627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:49.834087    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f05cda-6468ddbfc9-sbh9c"]
Oct 30 21:37:50.473585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:50.473548    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"
Oct 30 21:37:50.523159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:50.523118    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"]
Oct 30 21:37:50.533690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:50.533658    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f05cda-666854854c-vx8dm"]
Oct 30 21:37:51.214896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:51.214855    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab850567-4e74-48e9-9ff5-14795d4e0bc5" path="/var/lib/kubelet/pods/ab850567-4e74-48e9-9ff5-14795d4e0bc5/volumes"
Oct 30 21:37:51.215334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:51.215316    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26a5b26-e3bd-4250-974a-86400742b17b" path="/var/lib/kubelet/pods/d26a5b26-e3bd-4250-974a-86400742b17b/volumes"
Oct 30 21:37:51.620743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:51.620708    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:37:51.621179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:51.620781    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:54.507545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.507437    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"]
Oct 30 21:37:54.508129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508101    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d26a5b26-e3bd-4250-974a-86400742b17b" containerName="payload-pre-processing"
Oct 30 21:37:54.508129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508116    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26a5b26-e3bd-4250-974a-86400742b17b" containerName="payload-pre-processing"
Oct 30 21:37:54.508129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508129    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a5d996ad-2f54-462d-9b9c-026f7fb5d870" containerName="revoke-keys"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508134    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d996ad-2f54-462d-9b9c-026f7fb5d870" containerName="revoke-keys"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508153    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" containerName="payload-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508157    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" containerName="payload-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508182    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ab850567-4e74-48e9-9ff5-14795d4e0bc5" containerName="payload-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508189    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab850567-4e74-48e9-9ff5-14795d4e0bc5" containerName="payload-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508198    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4a08fad2-c55e-4560-b7b5-0eddf63903a5" containerName="payload-pre-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508203    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a08fad2-c55e-4560-b7b5-0eddf63903a5" containerName="payload-pre-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508218    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8b11200d-d525-4395-8fbc-220b70efaeea" containerName="payload-pre-processing"
Oct 30 21:37:54.508313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508222    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b11200d-d525-4395-8fbc-220b70efaeea" containerName="payload-pre-processing"
Oct 30 21:37:54.508830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508331    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b11200d-d525-4395-8fbc-220b70efaeea" containerName="payload-pre-processing"
Oct 30 21:37:54.508830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508353    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26a5b26-e3bd-4250-974a-86400742b17b" containerName="payload-pre-processing"
Oct 30 21:37:54.508830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508361    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d996ad-2f54-462d-9b9c-026f7fb5d870" containerName="revoke-keys"
Oct 30 21:37:54.508830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508371    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bb5c7f-d9f7-4c9b-a830-93d806eaaf4d" containerName="payload-processing"
Oct 30 21:37:54.508830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508384    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab850567-4e74-48e9-9ff5-14795d4e0bc5" containerName="payload-processing"
Oct 30 21:37:54.508830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.508396    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a08fad2-c55e-4560-b7b5-0eddf63903a5" containerName="payload-pre-processing"
Oct 30 21:37:54.510180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.510153    6695 generic.go:356] "Generic (PLEG): container finished" podID="d446414a-be00-4d2d-97cc-006645cb67d8" containerID="b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5" exitCode=0
Oct 30 21:37:54.512116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.512095    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" event={"ID":"d446414a-be00-4d2d-97cc-006645cb67d8","Type":"ContainerDied","Data":"b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"}
Oct 30 21:37:54.512243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.512227    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.512696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.512678    6695 scope.go:122] "RemoveContainer" containerID="b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"
Oct 30 21:37:54.514326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.514300    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:54.514415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.514360    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:54.514415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.514392    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:37:54.533404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.533378    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"]
Oct 30 21:37:54.560103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.560075    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4e4693\""
Oct 30 21:37:54.567593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.567475    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4e4693-dockercfg-rwhdh\""
Oct 30 21:37:54.567829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.567747    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.567939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.567882    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw648\" (UniqueName: \"kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648\") pod \"payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.670669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.670632    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.671068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.671045    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw648\" (UniqueName: \"kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648\") pod \"payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.683701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.672464    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.723918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.723632    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw648\" (UniqueName: \"kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648\") pod \"payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:54.758190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.758104    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"]
Oct 30 21:37:54.762944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.762908    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f05cda-2j2gx"]
Oct 30 21:37:54.834040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:54.834009    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:55.158008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.157975    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"]
Oct 30 21:37:55.161802 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:37:55.161773    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc0dcfc_dec0_4aa8_8e3b_29eb79d2fc5c.slice/crio-48ee3a86122284f814cf9983ceed7ffcac0ad9c7cdce055f1e0b9dd88f18d48a WatchSource:0}: Error finding container 48ee3a86122284f814cf9983ceed7ffcac0ad9c7cdce055f1e0b9dd88f18d48a: Status 404 returned error can't find the container with id 48ee3a86122284f814cf9983ceed7ffcac0ad9c7cdce055f1e0b9dd88f18d48a
Oct 30 21:37:55.216196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.216162    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d996ad-2f54-462d-9b9c-026f7fb5d870" path="/var/lib/kubelet/pods/a5d996ad-2f54-462d-9b9c-026f7fb5d870/volumes"
Oct 30 21:37:55.542676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.542639    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" event={"ID":"d446414a-be00-4d2d-97cc-006645cb67d8","Type":"ContainerStarted","Data":"5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0"}
Oct 30 21:37:55.544549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.544518    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k" event={"ID":"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c","Type":"ContainerStarted","Data":"436b4f95d036338c3ba6ff16ed16ae89a7e8fb152cd3e0f7ab38f8fcc36209dc"}
Oct 30 21:37:55.544716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.544568    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k" event={"ID":"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c","Type":"ContainerStarted","Data":"48ee3a86122284f814cf9983ceed7ffcac0ad9c7cdce055f1e0b9dd88f18d48a"}
Oct 30 21:37:55.544716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.544614    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:37:55.654631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.650788    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k" podStartSLOduration=1.650767742 podStartE2EDuration="1.650767742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:55.649176852 +0000 UTC m=+1658.571635151" watchObservedRunningTime="2026-10-30 21:37:55.650767742 +0000 UTC m=+1658.573226041"
Oct 30 21:37:55.705683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.705653    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:37:55.705980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.705950    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:37:55.706110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:55.705998    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:37:56.711031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:56.710992    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:37:56.711641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:56.711586    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:37:57.165747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.165716    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"]
Oct 30 21:37:57.175667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.175633    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-13658f-79x54"]
Oct 30 21:37:57.396633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.394751    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0551363-9b5d-4e57-965e-1eb78384ad53" path="/var/lib/kubelet/pods/e0551363-9b5d-4e57-965e-1eb78384ad53/volumes"
Oct 30 21:37:57.705763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.705729    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:37:57.705972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.705786    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:37:57.801814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.801782    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"]
Oct 30 21:37:57.804682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:57.804651    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-26a13f-nh42n"]
Oct 30 21:37:58.705908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:58.705870    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:37:58.706100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:58.705929    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:37:59.237477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:59.237433    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a5b4f5-6598-4d39-bc8a-6fb29b8eae08" path="/var/lib/kubelet/pods/15a5b4f5-6598-4d39-bc8a-6fb29b8eae08/volumes"
Oct 30 21:37:59.705474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:59.705441    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:37:59.705677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:37:59.705510    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:38:00.705584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:00.705546    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:38:00.706091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:00.705633    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:38:01.180628 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:01.177899    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:38:01.325816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:01.325772    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"]
Oct 30 21:38:01.326074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:01.326025    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy" containerID="cri-o://ad578183f56b5614e7eb2b86cb133105a68a14470bc2f95896bb24df82058cfb" gracePeriod=30
Oct 30 21:38:01.709879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:01.709847    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:38:01.710285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:01.710159    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:38:01.710976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:01.710955    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:38:03.361525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.361483    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9" podUID="a6ee646f-f360-4105-a775-a82d607efbf9" containerName="main" containerID="cri-o://f43129d133308c7ec30d7b105ac74e6fdffa258725753c70cf957a0ac7d104d2" gracePeriod=45
Oct 30 21:38:03.612920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.612835    6695 generic.go:356] "Generic (PLEG): container finished" podID="a6ee646f-f360-4105-a775-a82d607efbf9" containerID="f43129d133308c7ec30d7b105ac74e6fdffa258725753c70cf957a0ac7d104d2" exitCode=0
Oct 30 21:38:03.613061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.612914    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9" event={"ID":"a6ee646f-f360-4105-a775-a82d607efbf9","Type":"ContainerDied","Data":"f43129d133308c7ec30d7b105ac74e6fdffa258725753c70cf957a0ac7d104d2"}
Oct 30 21:38:03.686234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.686205    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:38:03.858668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.858637    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs\") pod \"a6ee646f-f360-4105-a775-a82d607efbf9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") "
Oct 30 21:38:03.858849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.858681    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir\") pod \"a6ee646f-f360-4105-a775-a82d607efbf9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") "
Oct 30 21:38:03.858849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.858747    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb\" (UniqueName: \"kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb\") pod \"a6ee646f-f360-4105-a775-a82d607efbf9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") "
Oct 30 21:38:03.858849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.858818    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm\") pod \"a6ee646f-f360-4105-a775-a82d607efbf9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") "
Oct 30 21:38:03.859039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.858866    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home\") pod \"a6ee646f-f360-4105-a775-a82d607efbf9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") "
Oct 30 21:38:03.859039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.858964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache\") pod \"a6ee646f-f360-4105-a775-a82d607efbf9\" (UID: \"a6ee646f-f360-4105-a775-a82d607efbf9\") "
Oct 30 21:38:03.859039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.859009    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir" pod "a6ee646f-f360-4105-a775-a82d607efbf9" (UID: "a6ee646f-f360-4105-a775-a82d607efbf9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:03.859229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.859198    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home" pod "a6ee646f-f360-4105-a775-a82d607efbf9" (UID: "a6ee646f-f360-4105-a775-a82d607efbf9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:03.859300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.859237    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache" pod "a6ee646f-f360-4105-a775-a82d607efbf9" (UID: "a6ee646f-f360-4105-a775-a82d607efbf9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:03.859575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.859555    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:03.859575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.859574    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:03.859774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.859583    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:03.861015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.860990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm" pod "a6ee646f-f360-4105-a775-a82d607efbf9" (UID: "a6ee646f-f360-4105-a775-a82d607efbf9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:03.861121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.861057    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs" pod "a6ee646f-f360-4105-a775-a82d607efbf9" (UID: "a6ee646f-f360-4105-a775-a82d607efbf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:03.861121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.861113    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb" pod "a6ee646f-f360-4105-a775-a82d607efbf9" (UID: "a6ee646f-f360-4105-a775-a82d607efbf9"). InnerVolumeSpecName "kube-api-access-j54bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:03.961841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.961438    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ee646f-f360-4105-a775-a82d607efbf9-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:03.961841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.961480    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j54bb\" (UniqueName: \"kubernetes.io/projected/a6ee646f-f360-4105-a775-a82d607efbf9-kube-api-access-j54bb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:03.961841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:03.961495    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ee646f-f360-4105-a775-a82d607efbf9-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:04.197031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.196941    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"]
Oct 30 21:38:04.197892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.197248    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy" containerID="cri-o://da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec" gracePeriod=30
Oct 30 21:38:04.622077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.622050    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"
Oct 30 21:38:04.622480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.622097    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9" event={"ID":"a6ee646f-f360-4105-a775-a82d607efbf9","Type":"ContainerDied","Data":"7baa8bf4697a8db67dd6d1cbb0708560ba4aed2856232be3ba23fbba8e6c93ab"}
Oct 30 21:38:04.622480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.622147    6695 scope.go:122] "RemoveContainer" containerID="f43129d133308c7ec30d7b105ac74e6fdffa258725753c70cf957a0ac7d104d2"
Oct 30 21:38:04.670840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.670813    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"]
Oct 30 21:38:04.676126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.676102    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0a1ee3/ipp-route-0a1ee3-kserve-d44874c5c-lkhg9"]
Oct 30 21:38:04.821716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.821684    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"]
Oct 30 21:38:04.822328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.822314    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a6ee646f-f360-4105-a775-a82d607efbf9" containerName="main"
Oct 30 21:38:04.822384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.822329    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ee646f-f360-4105-a775-a82d607efbf9" containerName="main"
Oct 30 21:38:04.822439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.822430    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ee646f-f360-4105-a775-a82d607efbf9" containerName="main"
Oct 30 21:38:04.827096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.827077    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.841952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.841926    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:04.841952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.841932    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:04.841952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.841935    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:04.842166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.841990    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:38:04.849430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.849400    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"]
Oct 30 21:38:04.878479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.878400    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmll5\" (UniqueName: \"kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.878661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.878542    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.878754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.878727    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.980261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.980229    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmll5\" (UniqueName: \"kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.980464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.980301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.980464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.980404    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.980982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.980959    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.982788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.982768    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:04.999085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:04.999059    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmll5\" (UniqueName: \"kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:05.142422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.142335    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:05.218735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.218268    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ee646f-f360-4105-a775-a82d607efbf9" path="/var/lib/kubelet/pods/a6ee646f-f360-4105-a775-a82d607efbf9/volumes"
Oct 30 21:38:05.354658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.354242    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"]
Oct 30 21:38:05.354898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.354870    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy" containerID="cri-o://b31ad7054105ed753e3c68e43da0ec6706b3f0673d7a4a46f993fd8a013e6a94" gracePeriod=30
Oct 30 21:38:05.450336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.450250    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"]
Oct 30 21:38:05.692626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.690914    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r" event={"ID":"cf40cdc9-5890-4e15-a74c-9ccd69c0709c","Type":"ContainerStarted","Data":"56936290297c6fe3005527327fc321bb6eab175bef140ddc189f1df36bcc07c5"}
Oct 30 21:38:05.951843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.951808    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"]
Oct 30 21:38:05.955901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.955883    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:05.974262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.974237    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"]
Oct 30 21:38:05.992541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.992508    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:05.992709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.992577    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjsn\" (UniqueName: \"kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:05.992805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:05.992742    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.014157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.014128    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"]
Oct 30 21:38:06.018580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.018559    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.037276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.037253    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"]
Oct 30 21:38:06.093493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.093454    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.093701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.093657    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.093764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.093700    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjsn\" (UniqueName: \"kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.093764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.093724    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn8d\" (UniqueName: \"kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.093764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.093754    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.093900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.093771    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.094293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.094274    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.096132 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.096114    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.111396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.111367    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjsn\" (UniqueName: \"kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn\") pod \"maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.195882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.195845    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.196082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.196031    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn8d\" (UniqueName: \"kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.196229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.196095    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.196764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.196743    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.199022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.198967    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.217488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.217463    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn8d\" (UniqueName: \"kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d\") pod \"maas-api-revoke-sub-b0e6ac19-jtszc\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.270007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.269976    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:06.333917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.333847    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:06.562223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.561421    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:38:06.589651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.588647    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"]
Oct 30 21:38:06.594966 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:06.594927    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c935ce_99ff_4538_b7dc_6a75a9b6dfc7.slice/crio-80643352e26358636176ea46f94c7aa93e4af336ce40a399b58724ae02c3ba5f WatchSource:0}: Error finding container 80643352e26358636176ea46f94c7aa93e4af336ce40a399b58724ae02c3ba5f: Status 404 returned error can't find the container with id 80643352e26358636176ea46f94c7aa93e4af336ce40a399b58724ae02c3ba5f
Oct 30 21:38:06.668690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.668653    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"]
Oct 30 21:38:06.738819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.738745    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc" event={"ID":"75866545-81f3-4335-abf8-3a2941aaaf06","Type":"ContainerStarted","Data":"9bf9a26b4af92af1145241b244685b4a90753e576f911ab4143f5c7b97485671"}
Oct 30 21:38:06.743308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.743281    6695 generic.go:356] "Generic (PLEG): container finished" podID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerID="ad578183f56b5614e7eb2b86cb133105a68a14470bc2f95896bb24df82058cfb" exitCode=0
Oct 30 21:38:06.744652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.743358    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" event={"ID":"221369ca-f9e3-4d12-8279-13e85182e7c9","Type":"ContainerDied","Data":"ad578183f56b5614e7eb2b86cb133105a68a14470bc2f95896bb24df82058cfb"}
Oct 30 21:38:06.744652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.743586    6695 scope.go:122] "RemoveContainer" containerID="16d05cc93d83599bd5c8430a943dc4423ab0a7f682e609c55428ce0131f82d0d"
Oct 30 21:38:06.748838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.748501    6695 generic.go:356] "Generic (PLEG): container finished" podID="cf40cdc9-5890-4e15-a74c-9ccd69c0709c" containerID="bfe556fec3a50346be6aa761910acc426204cf199461640f660e68eb60eba467" exitCode=0
Oct 30 21:38:06.748838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.748578    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r" event={"ID":"cf40cdc9-5890-4e15-a74c-9ccd69c0709c","Type":"ContainerDied","Data":"bfe556fec3a50346be6aa761910acc426204cf199461640f660e68eb60eba467"}
Oct 30 21:38:06.750196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.750171    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk" event={"ID":"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7","Type":"ContainerStarted","Data":"80643352e26358636176ea46f94c7aa93e4af336ce40a399b58724ae02c3ba5f"}
Oct 30 21:38:06.833932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.833903    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:38:06.926496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926461    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.926707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926523    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.926707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926562    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.926707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926671    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.926875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926711    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.926875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926725    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:06.926875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926753    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.927176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926862    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:06.927176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.926913    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.927291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.927260    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:06.927633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.927577    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:06.928640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.928579    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.928758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.928681    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2\") pod \"221369ca-f9e3-4d12-8279-13e85182e7c9\" (UID: \"221369ca-f9e3-4d12-8279-13e85182e7c9\") "
Oct 30 21:38:06.928919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.928900    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:06.929449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.929330    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.929449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.929359    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.929449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.929375    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.929449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.929391    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.929449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.929405    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221369ca-f9e3-4d12-8279-13e85182e7c9-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:06.929968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.929943    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:06.930488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:06.930466    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:07.011103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.011074    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:07.011218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.011199    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2" pod "221369ca-f9e3-4d12-8279-13e85182e7c9" (UID: "221369ca-f9e3-4d12-8279-13e85182e7c9"). InnerVolumeSpecName "kube-api-access-sj4h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:07.030220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.030197    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj4h2\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-kube-api-access-sj4h2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:07.030298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.030225    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:07.030298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.030234    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:07.030298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.030243    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221369ca-f9e3-4d12-8279-13e85182e7c9-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:07.758982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.758949    6695 generic.go:356] "Generic (PLEG): container finished" podID="26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" containerID="3d2edbbd328c9427e5bff3ee392e4fea2776120c76b0240ca3f63160ff4649f4" exitCode=0
Oct 30 21:38:07.759406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.759040    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk" event={"ID":"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7","Type":"ContainerDied","Data":"3d2edbbd328c9427e5bff3ee392e4fea2776120c76b0240ca3f63160ff4649f4"}
Oct 30 21:38:07.760721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.760697    6695 generic.go:356] "Generic (PLEG): container finished" podID="75866545-81f3-4335-abf8-3a2941aaaf06" containerID="eda473c0b8f3f8122fccc8b5d6a85a57d7f60dbb02544f10b37a2ba96a0145bd" exitCode=0
Oct 30 21:38:07.760871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.760777    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc" event={"ID":"75866545-81f3-4335-abf8-3a2941aaaf06","Type":"ContainerDied","Data":"eda473c0b8f3f8122fccc8b5d6a85a57d7f60dbb02544f10b37a2ba96a0145bd"}
Oct 30 21:38:07.762183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.762165    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"
Oct 30 21:38:07.762287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.762204    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224" event={"ID":"221369ca-f9e3-4d12-8279-13e85182e7c9","Type":"ContainerDied","Data":"e16ac5536c79278d03c8cda756f05542e767c848ccff55f176899238a808b6bd"}
Oct 30 21:38:07.762287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.762253    6695 scope.go:122] "RemoveContainer" containerID="ad578183f56b5614e7eb2b86cb133105a68a14470bc2f95896bb24df82058cfb"
Oct 30 21:38:07.874084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.874041    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"]
Oct 30 21:38:07.886222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.886191    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f05cda-openshift-default-796b546cfc-kw224"]
Oct 30 21:38:07.913162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.913136    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:07.942057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.942029    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca\") pod \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") "
Oct 30 21:38:07.942202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.942158    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca\") pod \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") "
Oct 30 21:38:07.942306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.942291    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5\" (UniqueName: \"kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5\") pod \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\" (UID: \"cf40cdc9-5890-4e15-a74c-9ccd69c0709c\") "
Oct 30 21:38:07.942421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.942396    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca" pod "cf40cdc9-5890-4e15-a74c-9ccd69c0709c" (UID: "cf40cdc9-5890-4e15-a74c-9ccd69c0709c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:07.942664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.942645    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:07.944091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.944073    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca" pod "cf40cdc9-5890-4e15-a74c-9ccd69c0709c" (UID: "cf40cdc9-5890-4e15-a74c-9ccd69c0709c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:07.944201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:07.944186    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5" pod "cf40cdc9-5890-4e15-a74c-9ccd69c0709c" (UID: "cf40cdc9-5890-4e15-a74c-9ccd69c0709c"). InnerVolumeSpecName "kube-api-access-pmll5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:08.044354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:08.044325    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:08.044354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:08.044351    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmll5\" (UniqueName: \"kubernetes.io/projected/cf40cdc9-5890-4e15-a74c-9ccd69c0709c-kube-api-access-pmll5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:08.776498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:08.776473    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"
Oct 30 21:38:08.776947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:08.776509    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r" event={"ID":"cf40cdc9-5890-4e15-a74c-9ccd69c0709c","Type":"ContainerDied","Data":"56936290297c6fe3005527327fc321bb6eab175bef140ddc189f1df36bcc07c5"}
Oct 30 21:38:08.776947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:08.776553    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56936290297c6fe3005527327fc321bb6eab175bef140ddc189f1df36bcc07c5"
Oct 30 21:38:09.013791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.013767    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:09.017116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.017094    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:09.056799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.056490    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn\" (UniqueName: \"kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn\") pod \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") "
Oct 30 21:38:09.056799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.056545    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca\") pod \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") "
Oct 30 21:38:09.056799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.056593    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca\") pod \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\" (UID: \"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7\") "
Oct 30 21:38:09.056799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.056657    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d\" (UniqueName: \"kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d\") pod \"75866545-81f3-4335-abf8-3a2941aaaf06\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") "
Oct 30 21:38:09.056799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.056676    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca\") pod \"75866545-81f3-4335-abf8-3a2941aaaf06\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") "
Oct 30 21:38:09.056799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.056703    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca\") pod \"75866545-81f3-4335-abf8-3a2941aaaf06\" (UID: \"75866545-81f3-4335-abf8-3a2941aaaf06\") "
Oct 30 21:38:09.057695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.057648    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca" pod "26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" (UID: "26c935ce-99ff-4538-b7dc-6a75a9b6dfc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:09.060486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.058023    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca" pod "75866545-81f3-4335-abf8-3a2941aaaf06" (UID: "75866545-81f3-4335-abf8-3a2941aaaf06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:09.060486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.060093    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca" pod "26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" (UID: "26c935ce-99ff-4538-b7dc-6a75a9b6dfc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:09.060939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.060919    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca" pod "75866545-81f3-4335-abf8-3a2941aaaf06" (UID: "75866545-81f3-4335-abf8-3a2941aaaf06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:09.061021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.060958    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn" pod "26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" (UID: "26c935ce-99ff-4538-b7dc-6a75a9b6dfc7"). InnerVolumeSpecName "kube-api-access-vfjsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:09.062244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.062224    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d" pod "75866545-81f3-4335-abf8-3a2941aaaf06" (UID: "75866545-81f3-4335-abf8-3a2941aaaf06"). InnerVolumeSpecName "kube-api-access-wxn8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:09.158955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.158927    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjsn\" (UniqueName: \"kubernetes.io/projected/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-kube-api-access-vfjsn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.158955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.158953    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.159159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.158964    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.159159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.158974    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxn8d\" (UniqueName: \"kubernetes.io/projected/75866545-81f3-4335-abf8-3a2941aaaf06-kube-api-access-wxn8d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.159159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.158984    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.159159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.158993    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75866545-81f3-4335-abf8-3a2941aaaf06-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.215044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.214928    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" path="/var/lib/kubelet/pods/221369ca-f9e3-4d12-8279-13e85182e7c9/volumes"
Oct 30 21:38:09.444269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.444242    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:38:09.461453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461429    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461485    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461527    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461556    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461580    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461712    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461741    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461787    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461815    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert\") pod \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\" (UID: \"b49aa851-f301-40f5-bc59-cc2fb9b80ccf\") "
Oct 30 21:38:09.461918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461874    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:09.462322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.461917    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:09.462322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462040    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:09.462322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462121    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:09.462562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462347    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:09.462562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462465    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.462562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462488    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.462562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462502    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.462562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462516    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.462562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.462531    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.464495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.464453    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "kube-api-access-kv87m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:09.464767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.464739    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:09.464865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.464775    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:09.464865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.464791    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token" pod "b49aa851-f301-40f5-bc59-cc2fb9b80ccf" (UID: "b49aa851-f301-40f5-bc59-cc2fb9b80ccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:09.563993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.563907    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv87m\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-kube-api-access-kv87m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.563993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.563935    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.563993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.563945    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.563993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.563953    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b49aa851-f301-40f5-bc59-cc2fb9b80ccf-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:09.785619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.785575    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"
Oct 30 21:38:09.786011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.785617    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk" event={"ID":"26c935ce-99ff-4538-b7dc-6a75a9b6dfc7","Type":"ContainerDied","Data":"80643352e26358636176ea46f94c7aa93e4af336ce40a399b58724ae02c3ba5f"}
Oct 30 21:38:09.786011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.785710    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80643352e26358636176ea46f94c7aa93e4af336ce40a399b58724ae02c3ba5f"
Oct 30 21:38:09.787198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.787176    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"
Oct 30 21:38:09.787352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.787229    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc" event={"ID":"75866545-81f3-4335-abf8-3a2941aaaf06","Type":"ContainerDied","Data":"9bf9a26b4af92af1145241b244685b4a90753e576f911ab4143f5c7b97485671"}
Oct 30 21:38:09.787352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.787274    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bf9a26b4af92af1145241b244685b4a90753e576f911ab4143f5c7b97485671"
Oct 30 21:38:09.789397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.789365    6695 generic.go:356] "Generic (PLEG): container finished" podID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerID="da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec" exitCode=0
Oct 30 21:38:09.789544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.789418    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"
Oct 30 21:38:09.789544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.789448    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" event={"ID":"b49aa851-f301-40f5-bc59-cc2fb9b80ccf","Type":"ContainerDied","Data":"da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec"}
Oct 30 21:38:09.789544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.789487    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48" event={"ID":"b49aa851-f301-40f5-bc59-cc2fb9b80ccf","Type":"ContainerDied","Data":"cfeea236d932cbd5b2a4f5195d0a00f122b4610526e66d3b022692e0e29e45b3"}
Oct 30 21:38:09.789544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.789507    6695 scope.go:122] "RemoveContainer" containerID="da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec"
Oct 30 21:38:09.800801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.800782    6695 scope.go:122] "RemoveContainer" containerID="acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"
Oct 30 21:38:09.810136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.810120    6695 scope.go:122] "RemoveContainer" containerID="da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec"
Oct 30 21:38:09.810403 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:09.810383    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec\": container with ID starting with da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec not found: ID does not exist" containerID="da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec"
Oct 30 21:38:09.810454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.810412    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec"} err="failed to get container status \"da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec\": rpc error: code = NotFound desc = could not find container \"da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec\": container with ID starting with da8bbbabb64188635271c3edf5f45777a4fc1c6eb581b50483c0cee94b4707ec not found: ID does not exist"
Oct 30 21:38:09.810454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.810431    6695 scope.go:122] "RemoveContainer" containerID="acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"
Oct 30 21:38:09.810738 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:09.810722    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc\": container with ID starting with acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc not found: ID does not exist" containerID="acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"
Oct 30 21:38:09.810813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.810742    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc"} err="failed to get container status \"acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc\": rpc error: code = NotFound desc = could not find container \"acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc\": container with ID starting with acf9de5c2f0c9d8202b7660fe10ccb355629131b276791b617962bae8380a6bc not found: ID does not exist"
Oct 30 21:38:09.842462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.842412    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"]
Oct 30 21:38:09.853258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:09.853238    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-13658f-openshift-default-5bcc4f4d7c-6td48"]
Oct 30 21:38:10.799566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:10.799534    6695 generic.go:356] "Generic (PLEG): container finished" podID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerID="b31ad7054105ed753e3c68e43da0ec6706b3f0673d7a4a46f993fd8a013e6a94" exitCode=0
Oct 30 21:38:10.800039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:10.799621    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" event={"ID":"37ac6495-1f88-4207-ad29-92fd7e2aba41","Type":"ContainerDied","Data":"b31ad7054105ed753e3c68e43da0ec6706b3f0673d7a4a46f993fd8a013e6a94"}
Oct 30 21:38:10.800039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:10.799674    6695 scope.go:122] "RemoveContainer" containerID="546ce07ba3010e65fd9fb89b79cf3cc4f930ae8801c4f0c7a61dac5535f84f4e"
Oct 30 21:38:11.038422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.038397    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:38:11.080905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.080817    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.080905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.080870    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.080965    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081076    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081118    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081164    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081189    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081236    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081325    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs\") pod \"37ac6495-1f88-4207-ad29-92fd7e2aba41\" (UID: \"37ac6495-1f88-4207-ad29-92fd7e2aba41\") "
Oct 30 21:38:11.081673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081648    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:11.081766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081737    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:11.081918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081898    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37ac6495-1f88-4207-ad29-92fd7e2aba41-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.081996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081924    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.081996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.081947    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:11.082160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.082140    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:11.082160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.082152    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:11.083973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.083948    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:11.084079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.084007    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:11.084079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.084015    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "kube-api-access-2wlqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:11.084079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.084034    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy" pod "37ac6495-1f88-4207-ad29-92fd7e2aba41" (UID: "37ac6495-1f88-4207-ad29-92fd7e2aba41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:11.183177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183143    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.183177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183173    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.183177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183182    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.183424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183190    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.183424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183202    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wlqq\" (UniqueName: \"kubernetes.io/projected/37ac6495-1f88-4207-ad29-92fd7e2aba41-kube-api-access-2wlqq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.183424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183212    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.183424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.183220    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37ac6495-1f88-4207-ad29-92fd7e2aba41-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:11.214517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.214485    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" path="/var/lib/kubelet/pods/b49aa851-f301-40f5-bc59-cc2fb9b80ccf/volumes"
Oct 30 21:38:11.807620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.807579    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"
Oct 30 21:38:11.808071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.807574    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5" event={"ID":"37ac6495-1f88-4207-ad29-92fd7e2aba41","Type":"ContainerDied","Data":"a10c1051c1104927b59cedec3558d64fde7f4644f6e3b1a323ba6aaf7aac8ad2"}
Oct 30 21:38:11.808071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.807747    6695 scope.go:122] "RemoveContainer" containerID="b31ad7054105ed753e3c68e43da0ec6706b3f0673d7a4a46f993fd8a013e6a94"
Oct 30 21:38:11.869250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.869214    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"]
Oct 30 21:38:11.878709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:11.878676    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-26a13f-openshift-default-566b5866fc-dzsk5"]
Oct 30 21:38:13.213461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:13.213430    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" path="/var/lib/kubelet/pods/37ac6495-1f88-4207-ad29-92fd7e2aba41/volumes"
Oct 30 21:38:15.302020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:15.301982    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"]
Oct 30 21:38:15.332596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:15.332563    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"]
Oct 30 21:38:16.482654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:16.482619    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"]
Oct 30 21:38:16.519255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:16.519222    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"]
Oct 30 21:38:19.375736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:19.375706    6695 scope.go:122] "RemoveContainer" containerID="d147681b4fbe4900f3a37cf93ba96f926197376ef4f86657146b9445d459692a"
Oct 30 21:38:20.302775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.302729    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd" podUID="ff444b02-6644-4b46-8872-d2c0f8de1afc" containerName="payload-processing" containerID="cri-o://4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a" gracePeriod=25
Oct 30 21:38:20.333565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.333515    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6" podUID="544ae917-21bc-4dac-9db3-218298b99122" containerName="payload-pre-processing" containerID="cri-o://b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5" gracePeriod=25
Oct 30 21:38:20.608617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.608578    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:38:20.732161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.728144    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:38:20.732828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.732569    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw\" (UniqueName: \"kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw\") pod \"ff444b02-6644-4b46-8872-d2c0f8de1afc\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") "
Oct 30 21:38:20.732828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.732808    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume\") pod \"ff444b02-6644-4b46-8872-d2c0f8de1afc\" (UID: \"ff444b02-6644-4b46-8872-d2c0f8de1afc\") "
Oct 30 21:38:20.733485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.733286    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume" pod "ff444b02-6644-4b46-8872-d2c0f8de1afc" (UID: "ff444b02-6644-4b46-8872-d2c0f8de1afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:20.733485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.733396    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff444b02-6644-4b46-8872-d2c0f8de1afc-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.737513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.737487    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw" pod "ff444b02-6644-4b46-8872-d2c0f8de1afc" (UID: "ff444b02-6644-4b46-8872-d2c0f8de1afc"). InnerVolumeSpecName "kube-api-access-kcblw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:20.834266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.834178    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume\") pod \"544ae917-21bc-4dac-9db3-218298b99122\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") "
Oct 30 21:38:20.834453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.834346    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm\" (UniqueName: \"kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm\") pod \"544ae917-21bc-4dac-9db3-218298b99122\" (UID: \"544ae917-21bc-4dac-9db3-218298b99122\") "
Oct 30 21:38:20.834811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.834590    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume" pod "544ae917-21bc-4dac-9db3-218298b99122" (UID: "544ae917-21bc-4dac-9db3-218298b99122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:20.834950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.834930    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcblw\" (UniqueName: \"kubernetes.io/projected/ff444b02-6644-4b46-8872-d2c0f8de1afc-kube-api-access-kcblw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.835196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.834957    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/544ae917-21bc-4dac-9db3-218298b99122-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.851944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.837880    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm" pod "544ae917-21bc-4dac-9db3-218298b99122" (UID: "544ae917-21bc-4dac-9db3-218298b99122"). InnerVolumeSpecName "kube-api-access-fmfjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:20.942236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.942191    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmfjm\" (UniqueName: \"kubernetes.io/projected/544ae917-21bc-4dac-9db3-218298b99122-kube-api-access-fmfjm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.970785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.970749    6695 generic.go:356] "Generic (PLEG): container finished" podID="ff444b02-6644-4b46-8872-d2c0f8de1afc" containerID="4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a" exitCode=0
Oct 30 21:38:20.970935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.970793    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"
Oct 30 21:38:20.970935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.970816    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd" event={"ID":"ff444b02-6644-4b46-8872-d2c0f8de1afc","Type":"ContainerDied","Data":"4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a"}
Oct 30 21:38:20.970935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.970900    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd" event={"ID":"ff444b02-6644-4b46-8872-d2c0f8de1afc","Type":"ContainerDied","Data":"f0466583e8a6bf1d38ab8b82b040a0d9e55521b62f014f8253be3a28695d0415"}
Oct 30 21:38:20.970935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.970928    6695 scope.go:122] "RemoveContainer" containerID="4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a"
Oct 30 21:38:20.973475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.973451    6695 generic.go:356] "Generic (PLEG): container finished" podID="544ae917-21bc-4dac-9db3-218298b99122" containerID="b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5" exitCode=0
Oct 30 21:38:20.973578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.973528    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6" event={"ID":"544ae917-21bc-4dac-9db3-218298b99122","Type":"ContainerDied","Data":"b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5"}
Oct 30 21:38:20.973578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.973559    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6" event={"ID":"544ae917-21bc-4dac-9db3-218298b99122","Type":"ContainerDied","Data":"677192cf5ce034e06e1582b6d98dab5bc9810e20371dbd129ad08b86ae993e7c"}
Oct 30 21:38:20.973705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.973695    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"
Oct 30 21:38:20.984691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.984667    6695 scope.go:122] "RemoveContainer" containerID="4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a"
Oct 30 21:38:20.984999 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:20.984974    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a\": container with ID starting with 4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a not found: ID does not exist" containerID="4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a"
Oct 30 21:38:20.985080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.985008    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a"} err="failed to get container status \"4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a\": rpc error: code = NotFound desc = could not find container \"4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a\": container with ID starting with 4e6ffb8237235764e6c2c194d9252d5e5e7e56e3e058425fa7c23915d511a41a not found: ID does not exist"
Oct 30 21:38:20.985080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.985031    6695 scope.go:122] "RemoveContainer" containerID="b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5"
Oct 30 21:38:20.994096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.994077    6695 scope.go:122] "RemoveContainer" containerID="b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5"
Oct 30 21:38:20.994345 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:20.994324    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5\": container with ID starting with b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5 not found: ID does not exist" containerID="b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5"
Oct 30 21:38:20.994417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:20.994351    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5"} err="failed to get container status \"b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5\": rpc error: code = NotFound desc = could not find container \"b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5\": container with ID starting with b2d2974b74bea722195cc27d72cf5f6d87c6b548cbb514c0801724e5210645e5 not found: ID does not exist"
Oct 30 21:38:21.018945 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.018921    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"]
Oct 30 21:38:21.029884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.029859    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea45c0-7cdf9dcb97-45tsd"]
Oct 30 21:38:21.055255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.055225    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"]
Oct 30 21:38:21.062257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.062235    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea45c0-6bb5985575-j2hz6"]
Oct 30 21:38:21.126947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.126864    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"]
Oct 30 21:38:21.129960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.129929    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea45c0-4xw6r"]
Oct 30 21:38:21.232767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.232196    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544ae917-21bc-4dac-9db3-218298b99122" path="/var/lib/kubelet/pods/544ae917-21bc-4dac-9db3-218298b99122/volumes"
Oct 30 21:38:21.233752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.233697    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf40cdc9-5890-4e15-a74c-9ccd69c0709c" path="/var/lib/kubelet/pods/cf40cdc9-5890-4e15-a74c-9ccd69c0709c/volumes"
Oct 30 21:38:21.234913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.234266    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff444b02-6644-4b46-8872-d2c0f8de1afc" path="/var/lib/kubelet/pods/ff444b02-6644-4b46-8872-d2c0f8de1afc/volumes"
Oct 30 21:38:21.482964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.482869    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2" podUID="7283c843-f443-427c-ac30-825416123201" containerName="payload-processing" containerID="cri-o://245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23" gracePeriod=25
Oct 30 21:38:21.520497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.520454    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h" podUID="3bfef07d-e40c-4069-8288-932dfc99f368" containerName="payload-pre-processing" containerID="cri-o://9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63" gracePeriod=25
Oct 30 21:38:21.801446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.801419    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:38:21.806627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.806595    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:38:21.956264 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.956228    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume\") pod \"7283c843-f443-427c-ac30-825416123201\" (UID: \"7283c843-f443-427c-ac30-825416123201\") "
Oct 30 21:38:21.956472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.956366    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume\") pod \"3bfef07d-e40c-4069-8288-932dfc99f368\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") "
Oct 30 21:38:21.956472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.956431    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc\" (UniqueName: \"kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc\") pod \"7283c843-f443-427c-ac30-825416123201\" (UID: \"7283c843-f443-427c-ac30-825416123201\") "
Oct 30 21:38:21.956642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.956512    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw\" (UniqueName: \"kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw\") pod \"3bfef07d-e40c-4069-8288-932dfc99f368\" (UID: \"3bfef07d-e40c-4069-8288-932dfc99f368\") "
Oct 30 21:38:21.956721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.956664    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume" pod "7283c843-f443-427c-ac30-825416123201" (UID: "7283c843-f443-427c-ac30-825416123201"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:21.956906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.956873    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume" pod "3bfef07d-e40c-4069-8288-932dfc99f368" (UID: "3bfef07d-e40c-4069-8288-932dfc99f368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:21.957119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.957103    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7283c843-f443-427c-ac30-825416123201-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:21.957175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.957126    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bfef07d-e40c-4069-8288-932dfc99f368-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:21.958632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.958596    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw" pod "3bfef07d-e40c-4069-8288-932dfc99f368" (UID: "3bfef07d-e40c-4069-8288-932dfc99f368"). InnerVolumeSpecName "kube-api-access-h2vqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:21.958731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.958642    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc" pod "7283c843-f443-427c-ac30-825416123201" (UID: "7283c843-f443-427c-ac30-825416123201"). InnerVolumeSpecName "kube-api-access-w4jrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:21.982409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.982380    6695 generic.go:356] "Generic (PLEG): container finished" podID="3bfef07d-e40c-4069-8288-932dfc99f368" containerID="9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63" exitCode=0
Oct 30 21:38:21.982569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.982426    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"
Oct 30 21:38:21.982569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.982465    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h" event={"ID":"3bfef07d-e40c-4069-8288-932dfc99f368","Type":"ContainerDied","Data":"9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63"}
Oct 30 21:38:21.982569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.982523    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h" event={"ID":"3bfef07d-e40c-4069-8288-932dfc99f368","Type":"ContainerDied","Data":"41fcca3b4895fc8e75ce6411eeafd603e2a7369d6bb2ed6d79c23f01c2e0c396"}
Oct 30 21:38:21.982569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.982559    6695 scope.go:122] "RemoveContainer" containerID="9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63"
Oct 30 21:38:21.983827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.983804    6695 generic.go:356] "Generic (PLEG): container finished" podID="7283c843-f443-427c-ac30-825416123201" containerID="245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23" exitCode=0
Oct 30 21:38:21.983942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.983851    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"
Oct 30 21:38:21.983942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.983888    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2" event={"ID":"7283c843-f443-427c-ac30-825416123201","Type":"ContainerDied","Data":"245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23"}
Oct 30 21:38:21.983942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.983930    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2" event={"ID":"7283c843-f443-427c-ac30-825416123201","Type":"ContainerDied","Data":"dc05a386694e6c6e9d9647a7c4100973a83026a12fc16fac5e4982a7ea659cef"}
Oct 30 21:38:21.994426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.994408    6695 scope.go:122] "RemoveContainer" containerID="9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63"
Oct 30 21:38:21.994747 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:21.994727    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63\": container with ID starting with 9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63 not found: ID does not exist" containerID="9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63"
Oct 30 21:38:21.994827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.994756    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63"} err="failed to get container status \"9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63\": rpc error: code = NotFound desc = could not find container \"9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63\": container with ID starting with 9eb9069269e95eac6a87b8825132623bf9ff11cd14572a290ee88bf4e8340e63 not found: ID does not exist"
Oct 30 21:38:21.994827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:21.994779    6695 scope.go:122] "RemoveContainer" containerID="245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23"
Oct 30 21:38:22.004161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.004143    6695 scope.go:122] "RemoveContainer" containerID="245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23"
Oct 30 21:38:22.004405 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:22.004386    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23\": container with ID starting with 245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23 not found: ID does not exist" containerID="245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23"
Oct 30 21:38:22.004467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.004413    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23"} err="failed to get container status \"245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23\": rpc error: code = NotFound desc = could not find container \"245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23\": container with ID starting with 245cc67501a0154800278a9bdfba9b9fb9e46e5d391cd3d02842d6b02af7bd23 not found: ID does not exist"
Oct 30 21:38:22.034397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.034320    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"]
Oct 30 21:38:22.037945 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.037921    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8a20c6-6489b897f5-lk55h"]
Oct 30 21:38:22.058029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.058002    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4jrc\" (UniqueName: \"kubernetes.io/projected/7283c843-f443-427c-ac30-825416123201-kube-api-access-w4jrc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:22.058029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.058026    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2vqw\" (UniqueName: \"kubernetes.io/projected/3bfef07d-e40c-4069-8288-932dfc99f368-kube-api-access-h2vqw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:22.069820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.069787    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"]
Oct 30 21:38:22.073189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.073159    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8a20c6-6d645fc6d5-gv5h2"]
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.945213    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"]
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947853    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947878    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947903    6695 cpu_manager.go:414] "containerMap: removing container" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947910    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947931    6695 cpu_manager.go:414] "containerMap: removing container" podUID="544ae917-21bc-4dac-9db3-218298b99122" containerName="payload-pre-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947941    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544ae917-21bc-4dac-9db3-218298b99122" containerName="payload-pre-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947964    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947971    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947980    6695 cpu_manager.go:414] "containerMap: removing container" podUID="26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947987    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.947997    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3bfef07d-e40c-4069-8288-932dfc99f368" containerName="payload-pre-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948004    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfef07d-e40c-4069-8288-932dfc99f368" containerName="payload-pre-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948020    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cf40cdc9-5890-4e15-a74c-9ccd69c0709c" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948027    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf40cdc9-5890-4e15-a74c-9ccd69c0709c" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948041    6695 cpu_manager.go:414] "containerMap: removing container" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948048    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948064    6695 cpu_manager.go:414] "containerMap: removing container" podUID="75866545-81f3-4335-abf8-3a2941aaaf06" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948070    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75866545-81f3-4335-abf8-3a2941aaaf06" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948084    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ff444b02-6644-4b46-8872-d2c0f8de1afc" containerName="payload-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948091    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff444b02-6644-4b46-8872-d2c0f8de1afc" containerName="payload-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948119    6695 cpu_manager.go:414] "containerMap: removing container" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948126    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948136    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7283c843-f443-427c-ac30-825416123201" containerName="payload-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948144    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7283c843-f443-427c-ac30-825416123201" containerName="payload-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948334    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948351    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="544ae917-21bc-4dac-9db3-218298b99122" containerName="payload-pre-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948361    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948370    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948378    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfef07d-e40c-4069-8288-932dfc99f368" containerName="payload-pre-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948391    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf40cdc9-5890-4e15-a74c-9ccd69c0709c" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948400    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948408    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="75866545-81f3-4335-abf8-3a2941aaaf06" containerName="revoke-keys"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948417    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff444b02-6644-4b46-8872-d2c0f8de1afc" containerName="payload-processing"
Oct 30 21:38:22.950665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.948429    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7283c843-f443-427c-ac30-825416123201" containerName="payload-processing"
Oct 30 21:38:22.959278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.959242    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"]
Oct 30 21:38:22.959442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.959277    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:22.973061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.973031    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:22.973061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.973053    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:22.973246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.973075    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:22.973246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:22.973084    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:38:23.073443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.073408    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.073670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.073461    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qrv\" (UniqueName: \"kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.073670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.073639    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.175144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.175108    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.175319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.175218    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.175319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.175252    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qrv\" (UniqueName: \"kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.175874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.175847    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.177468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.177448    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.193596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.193567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qrv\" (UniqueName: \"kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv\") pod \"maas-api-revoke-sub-71d6d355-zr7q4\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.213247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.213196    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfef07d-e40c-4069-8288-932dfc99f368" path="/var/lib/kubelet/pods/3bfef07d-e40c-4069-8288-932dfc99f368/volumes"
Oct 30 21:38:23.213585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.213572    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7283c843-f443-427c-ac30-825416123201" path="/var/lib/kubelet/pods/7283c843-f443-427c-ac30-825416123201/volumes"
Oct 30 21:38:23.274794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.274760    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:23.580991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:23.579876    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"]
Oct 30 21:38:23.585498 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:23.585464    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb07dd3_b043_4400_ac5d_df0c69b3d2eb.slice/crio-0814c799301bd23db7f2d65d0b45a102ec3f2847b777a760d0a1dcc55323dc47 WatchSource:0}: Error finding container 0814c799301bd23db7f2d65d0b45a102ec3f2847b777a760d0a1dcc55323dc47: Status 404 returned error can't find the container with id 0814c799301bd23db7f2d65d0b45a102ec3f2847b777a760d0a1dcc55323dc47
Oct 30 21:38:24.006055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:24.006025    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4" event={"ID":"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb","Type":"ContainerStarted","Data":"f396ba9ea73dc06d9cd735d33ab0b937c278bc2b4fae3dc753a4564bc47cd3f2"}
Oct 30 21:38:24.006468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:24.006078    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4" event={"ID":"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb","Type":"ContainerStarted","Data":"0814c799301bd23db7f2d65d0b45a102ec3f2847b777a760d0a1dcc55323dc47"}
Oct 30 21:38:24.046071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:24.046012    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4" podStartSLOduration=2.045997758 podStartE2EDuration="2.045997758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:24.042300134 +0000 UTC m=+1686.964758429" watchObservedRunningTime="2026-10-30 21:38:24.045997758 +0000 UTC m=+1686.968456056"
Oct 30 21:38:24.916644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:24.916541    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"]
Oct 30 21:38:24.919070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:24.919043    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8a20c6-gpzpk"]
Oct 30 21:38:25.015997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:25.015960    6695 generic.go:356] "Generic (PLEG): container finished" podID="7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" containerID="f396ba9ea73dc06d9cd735d33ab0b937c278bc2b4fae3dc753a4564bc47cd3f2" exitCode=0
Oct 30 21:38:25.016393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:25.016042    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4" event={"ID":"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb","Type":"ContainerDied","Data":"f396ba9ea73dc06d9cd735d33ab0b937c278bc2b4fae3dc753a4564bc47cd3f2"}
Oct 30 21:38:25.214014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:25.213934    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c935ce-99ff-4538-b7dc-6a75a9b6dfc7" path="/var/lib/kubelet/pods/26c935ce-99ff-4538-b7dc-6a75a9b6dfc7/volumes"
Oct 30 21:38:26.175561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.175526    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:26.329815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.329733    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca\") pod \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") "
Oct 30 21:38:26.329977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.329834    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca\") pod \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") "
Oct 30 21:38:26.329977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.329881    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv\" (UniqueName: \"kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv\") pod \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\" (UID: \"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb\") "
Oct 30 21:38:26.330144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.330119    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca" pod "7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" (UID: "7bb07dd3-b043-4400-ac5d-df0c69b3d2eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:26.330642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.330588    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:26.332081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.332054    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca" pod "7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" (UID: "7bb07dd3-b043-4400-ac5d-df0c69b3d2eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:26.332177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.332108    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv" pod "7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" (UID: "7bb07dd3-b043-4400-ac5d-df0c69b3d2eb"). InnerVolumeSpecName "kube-api-access-l7qrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:26.431785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.431747    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qrv\" (UniqueName: \"kubernetes.io/projected/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-kube-api-access-l7qrv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:26.431785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:26.431775    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:27.031501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:27.031470    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"
Oct 30 21:38:27.031688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:27.031501    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4" event={"ID":"7bb07dd3-b043-4400-ac5d-df0c69b3d2eb","Type":"ContainerDied","Data":"0814c799301bd23db7f2d65d0b45a102ec3f2847b777a760d0a1dcc55323dc47"}
Oct 30 21:38:27.031688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:27.031552    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0814c799301bd23db7f2d65d0b45a102ec3f2847b777a760d0a1dcc55323dc47"
Oct 30 21:38:28.231638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:28.231564    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"]
Oct 30 21:38:28.232120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:28.231853    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy" containerID="cri-o://31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5" gracePeriod=30
Oct 30 21:38:31.083862 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:31.083828    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:38:31.187048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.187013    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"]
Oct 30 21:38:31.187967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.187939    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" containerName="revoke-keys"
Oct 30 21:38:31.187967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.187965    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" containerName="revoke-keys"
Oct 30 21:38:31.188130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.187996    6695 cpu_manager.go:414] "containerMap: removing container" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy"
Oct 30 21:38:31.188130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.188003    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy"
Oct 30 21:38:31.188130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.188126    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="221369ca-f9e3-4d12-8279-13e85182e7c9" containerName="istio-proxy"
Oct 30 21:38:31.188294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.188144    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49aa851-f301-40f5-bc59-cc2fb9b80ccf" containerName="istio-proxy"
Oct 30 21:38:31.188294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.188158    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ac6495-1f88-4207-ad29-92fd7e2aba41" containerName="istio-proxy"
Oct 30 21:38:31.188294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.188175    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" containerName="revoke-keys"
Oct 30 21:38:31.191890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.191868    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.206551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.206523    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:31.218973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.218946    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:31.225021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.221743    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:31.225021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.223040    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"]
Oct 30 21:38:31.226857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.226835    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:38:31.280991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.280952    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.281190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.281025    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgxt\" (UniqueName: \"kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.281254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.281178    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.382673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.382576    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.382854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.382833    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.382915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.382888    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgxt\" (UniqueName: \"kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.383417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.383397    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.384879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.384852    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.404809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.404780    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgxt\" (UniqueName: \"kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt\") pod \"maas-api-revoke-sub-a0999ea2-6fhhh\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.506282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.506254    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:31.810685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.810652    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"]
Oct 30 21:38:31.814526 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:31.814498    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd495e107_386e_47ec_8cf6_6efaa3c6a5c3.slice/crio-68a3bbbc4a1a2d7d0c564b92fbf1b7145308669b517f2ce5984698d7db112937 WatchSource:0}: Error finding container 68a3bbbc4a1a2d7d0c564b92fbf1b7145308669b517f2ce5984698d7db112937: Status 404 returned error can't find the container with id 68a3bbbc4a1a2d7d0c564b92fbf1b7145308669b517f2ce5984698d7db112937
Oct 30 21:38:31.973545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.973506    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"]
Oct 30 21:38:31.973866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:31.973773    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" containerID="cri-o://ca2546b0a76f691ffcdf89a9237b374b1d891bd6b9d012fd555a69a46df0474b" gracePeriod=30
Oct 30 21:38:32.090478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:32.090373    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh" event={"ID":"d495e107-386e-47ec-8cf6-6efaa3c6a5c3","Type":"ContainerStarted","Data":"68a3bbbc4a1a2d7d0c564b92fbf1b7145308669b517f2ce5984698d7db112937"}
Oct 30 21:38:32.397254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:32.397175    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"]
Oct 30 21:38:32.397459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:32.397428    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" containerID="cri-o://06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075" gracePeriod=30
Oct 30 21:38:33.011074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.010983    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"]
Oct 30 21:38:33.015378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.015358    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.034595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.034570    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"]
Oct 30 21:38:33.098545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.098507    6695 generic.go:356] "Generic (PLEG): container finished" podID="d495e107-386e-47ec-8cf6-6efaa3c6a5c3" containerID="0274295a512395494787220e2e8ddfcdf4b9859b7a746bda0e09b1d3b5beb4c9" exitCode=0
Oct 30 21:38:33.098976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.098579    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh" event={"ID":"d495e107-386e-47ec-8cf6-6efaa3c6a5c3","Type":"ContainerDied","Data":"0274295a512395494787220e2e8ddfcdf4b9859b7a746bda0e09b1d3b5beb4c9"}
Oct 30 21:38:33.115933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.115900    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.116074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.115938    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.116074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.116009    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9pk\" (UniqueName: \"kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.216842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.216811    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.216842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.216844    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.217056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.216895    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9pk\" (UniqueName: \"kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.217523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.217495    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.219295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.219272    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.240158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.240124    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9pk\" (UniqueName: \"kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.329176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.329146    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:33.581182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.581122    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:38:33.641635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.641580    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"]
Oct 30 21:38:33.643576 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:33.643547    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d08feee_87ea_4173_8c91_1ee95524520e.slice/crio-8bb2f7d1ac76b9135ae1d88100463c93c8d112498b14cbd03027b80b559ea04a WatchSource:0}: Error finding container 8bb2f7d1ac76b9135ae1d88100463c93c8d112498b14cbd03027b80b559ea04a: Status 404 returned error can't find the container with id 8bb2f7d1ac76b9135ae1d88100463c93c8d112498b14cbd03027b80b559ea04a
Oct 30 21:38:33.728373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728350    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728408    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728437    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728477    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728578    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728667    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728729    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728816    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728850    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token\") pod \"c482d66f-1a6c-4407-8c32-20b4fb649d28\" (UID: \"c482d66f-1a6c-4407-8c32-20b4fb649d28\") "
Oct 30 21:38:33.728905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728847    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:33.729008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:33.729008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.728974    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:33.729128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.729085    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:33.729478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.729451    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.729646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.729484    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c482d66f-1a6c-4407-8c32-20b4fb649d28-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.729646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.729500    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.729646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.729515    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.729848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.729656    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:33.730497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.730454    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:33.730695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.730673    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "kube-api-access-qm8b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:33.730882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.730862    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:33.731080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.731063    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token" pod "c482d66f-1a6c-4407-8c32-20b4fb649d28" (UID: "c482d66f-1a6c-4407-8c32-20b4fb649d28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:33.831230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.831165    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.831230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.831193    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.831230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.831202    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.831230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.831212    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm8b9\" (UniqueName: \"kubernetes.io/projected/c482d66f-1a6c-4407-8c32-20b4fb649d28-kube-api-access-qm8b9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:33.831230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:33.831226    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c482d66f-1a6c-4407-8c32-20b4fb649d28-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:34.107320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.107212    6695 generic.go:356] "Generic (PLEG): container finished" podID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerID="31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5" exitCode=0
Oct 30 21:38:34.107320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.107276    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"
Oct 30 21:38:34.107320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.107276    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" event={"ID":"c482d66f-1a6c-4407-8c32-20b4fb649d28","Type":"ContainerDied","Data":"31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5"}
Oct 30 21:38:34.107918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.107353    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h" event={"ID":"c482d66f-1a6c-4407-8c32-20b4fb649d28","Type":"ContainerDied","Data":"daf6d990e7992fd490ae48725bc1f2ef2b8f52f19784d6f4583e4e9068da4897"}
Oct 30 21:38:34.107918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.107383    6695 scope.go:122] "RemoveContainer" containerID="31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5"
Oct 30 21:38:34.109862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.109821    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg" event={"ID":"8d08feee-87ea-4173-8c91-1ee95524520e","Type":"ContainerStarted","Data":"0f3a736b4ec7321c281047598158e12504d2fef65abf788bdffba082a13ea6e4"}
Oct 30 21:38:34.110019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.109901    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg" event={"ID":"8d08feee-87ea-4173-8c91-1ee95524520e","Type":"ContainerStarted","Data":"8bb2f7d1ac76b9135ae1d88100463c93c8d112498b14cbd03027b80b559ea04a"}
Oct 30 21:38:34.121754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.121732    6695 scope.go:122] "RemoveContainer" containerID="2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"
Oct 30 21:38:34.132875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.132796    6695 scope.go:122] "RemoveContainer" containerID="31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5"
Oct 30 21:38:34.133137 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:34.133105    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5\": container with ID starting with 31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5 not found: ID does not exist" containerID="31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5"
Oct 30 21:38:34.133257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.133145    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5"} err="failed to get container status \"31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5\": rpc error: code = NotFound desc = could not find container \"31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5\": container with ID starting with 31d645fc88266a35906e82f873513e7656a54507825555b00ab5abd7199cf4c5 not found: ID does not exist"
Oct 30 21:38:34.133257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.133166    6695 scope.go:122] "RemoveContainer" containerID="2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"
Oct 30 21:38:34.133472 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:34.133455    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2\": container with ID starting with 2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2 not found: ID does not exist" containerID="2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"
Oct 30 21:38:34.133523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.133485    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2"} err="failed to get container status \"2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2\": rpc error: code = NotFound desc = could not find container \"2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2\": container with ID starting with 2dd21275870b0a331a2ef3ba1f4a5dffe8a7628eaf2e532712e31e29283491b2 not found: ID does not exist"
Oct 30 21:38:34.164898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.164843    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg" podStartSLOduration=2.164828929 podStartE2EDuration="2.164828929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:34.162079261 +0000 UTC m=+1697.084537571" watchObservedRunningTime="2026-10-30 21:38:34.164828929 +0000 UTC m=+1697.087287227"
Oct 30 21:38:34.205297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.205261    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"]
Oct 30 21:38:34.213901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.213868    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea45c0-openshift-default-65fb98d9c-kl54h"]
Oct 30 21:38:34.238773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.238749    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:34.338098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.338068    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt\" (UniqueName: \"kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt\") pod \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") "
Oct 30 21:38:34.338258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.338247    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca\") pod \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") "
Oct 30 21:38:34.338306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.338289    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca\") pod \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\" (UID: \"d495e107-386e-47ec-8cf6-6efaa3c6a5c3\") "
Oct 30 21:38:34.338738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.338706    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca" pod "d495e107-386e-47ec-8cf6-6efaa3c6a5c3" (UID: "d495e107-386e-47ec-8cf6-6efaa3c6a5c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:34.340061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.340039    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt" pod "d495e107-386e-47ec-8cf6-6efaa3c6a5c3" (UID: "d495e107-386e-47ec-8cf6-6efaa3c6a5c3"). InnerVolumeSpecName "kube-api-access-fpgxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:34.340192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.340178    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca" pod "d495e107-386e-47ec-8cf6-6efaa3c6a5c3" (UID: "d495e107-386e-47ec-8cf6-6efaa3c6a5c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:34.440017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.439985    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpgxt\" (UniqueName: \"kubernetes.io/projected/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-kube-api-access-fpgxt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:34.440017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.440013    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:34.440017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:34.440022    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d495e107-386e-47ec-8cf6-6efaa3c6a5c3-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:35.118782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.118748    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"]
Oct 30 21:38:35.119362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119342    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"
Oct 30 21:38:35.119756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119732    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy"
Oct 30 21:38:35.119865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119756    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy"
Oct 30 21:38:35.119865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119770    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d495e107-386e-47ec-8cf6-6efaa3c6a5c3" containerName="revoke-keys"
Oct 30 21:38:35.119865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119778    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d495e107-386e-47ec-8cf6-6efaa3c6a5c3" containerName="revoke-keys"
Oct 30 21:38:35.120040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119979    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy"
Oct 30 21:38:35.120040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.119993    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d495e107-386e-47ec-8cf6-6efaa3c6a5c3" containerName="revoke-keys"
Oct 30 21:38:35.120040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.120005    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy"
Oct 30 21:38:35.122909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.122878    6695 generic.go:356] "Generic (PLEG): container finished" podID="8d08feee-87ea-4173-8c91-1ee95524520e" containerID="0f3a736b4ec7321c281047598158e12504d2fef65abf788bdffba082a13ea6e4" exitCode=0
Oct 30 21:38:35.123688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.123666    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh" event={"ID":"d495e107-386e-47ec-8cf6-6efaa3c6a5c3","Type":"ContainerDied","Data":"68a3bbbc4a1a2d7d0c564b92fbf1b7145308669b517f2ce5984698d7db112937"}
Oct 30 21:38:35.123783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.123703    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68a3bbbc4a1a2d7d0c564b92fbf1b7145308669b517f2ce5984698d7db112937"
Oct 30 21:38:35.123783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.123720    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg" event={"ID":"8d08feee-87ea-4173-8c91-1ee95524520e","Type":"ContainerDied","Data":"0f3a736b4ec7321c281047598158e12504d2fef65abf788bdffba082a13ea6e4"}
Oct 30 21:38:35.123858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.123795    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.159525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.159490    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"]
Oct 30 21:38:35.214520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.214476    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" path="/var/lib/kubelet/pods/c482d66f-1a6c-4407-8c32-20b4fb649d28/volumes"
Oct 30 21:38:35.253885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.253854    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.254053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.253925    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xqv\" (UniqueName: \"kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.254097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.254072    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.355569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.355532    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xqv\" (UniqueName: \"kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.356001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.355678    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.356001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.355787    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.356472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.356453    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.358159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.358141    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.375263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.375198    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xqv\" (UniqueName: \"kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv\") pod \"maas-api-revoke-sub-557e1bee-vcx8d\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.439078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.439039    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:35.535369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.535329    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"]
Oct 30 21:38:35.535993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.535971    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy"
Oct 30 21:38:35.535993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.535993    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c482d66f-1a6c-4407-8c32-20b4fb649d28" containerName="istio-proxy"
Oct 30 21:38:35.541884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.541858    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.559559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.559529    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"]
Oct 30 21:38:35.659726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.659644    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l579l\" (UniqueName: \"kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.659870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.659764    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.659870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.659842    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.719204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.719167    6695 patch_prober.go:29] interesting pod/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:38:35.719358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.719221    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:38:35.755277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.755222    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"]
Oct 30 21:38:35.779629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.761759    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.779629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.761969    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.779629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.762174    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l579l\" (UniqueName: \"kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.779629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.765041    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.779629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.773231    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.791923 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:35.786332    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751c146a_33a9_4f3b_b924_3d6ad36e32ad.slice/crio-dfc120f3042973f90a73bbcb043d95a1364063e0d493786375a0f39a9c5dcb5d WatchSource:0}: Error finding container dfc120f3042973f90a73bbcb043d95a1364063e0d493786375a0f39a9c5dcb5d: Status 404 returned error can't find the container with id dfc120f3042973f90a73bbcb043d95a1364063e0d493786375a0f39a9c5dcb5d
Oct 30 21:38:35.800448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.800364    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l579l\" (UniqueName: \"kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l\") pod \"maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:35.857836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:35.857809    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:36.132745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.132714    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d" event={"ID":"751c146a-33a9-4f3b-b924-3d6ad36e32ad","Type":"ContainerStarted","Data":"dfc120f3042973f90a73bbcb043d95a1364063e0d493786375a0f39a9c5dcb5d"}
Oct 30 21:38:36.178928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.178845    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"]
Oct 30 21:38:36.183697 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:36.183348    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb775af41_77b2_4daf_9b6b_c7097282db85.slice/crio-58f5af7a09425589796436bcb73e4e7f593cf4b13dadce19d79ade4ae1f05d29 WatchSource:0}: Error finding container 58f5af7a09425589796436bcb73e4e7f593cf4b13dadce19d79ade4ae1f05d29: Status 404 returned error can't find the container with id 58f5af7a09425589796436bcb73e4e7f593cf4b13dadce19d79ade4ae1f05d29
Oct 30 21:38:36.187283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.187232    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d" podStartSLOduration=1.18721766 podStartE2EDuration="1.18721766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:36.183130691 +0000 UTC m=+1699.105588989" watchObservedRunningTime="2026-10-30 21:38:36.18721766 +0000 UTC m=+1699.109675973"
Oct 30 21:38:36.263918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.263891    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:36.368947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.368908    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk\" (UniqueName: \"kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk\") pod \"8d08feee-87ea-4173-8c91-1ee95524520e\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") "
Oct 30 21:38:36.369108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.368961    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca\") pod \"8d08feee-87ea-4173-8c91-1ee95524520e\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") "
Oct 30 21:38:36.369108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.369042    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca\") pod \"8d08feee-87ea-4173-8c91-1ee95524520e\" (UID: \"8d08feee-87ea-4173-8c91-1ee95524520e\") "
Oct 30 21:38:36.369353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.369324    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca" pod "8d08feee-87ea-4173-8c91-1ee95524520e" (UID: "8d08feee-87ea-4173-8c91-1ee95524520e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:36.369540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.369511    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:36.371143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.371124    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk" pod "8d08feee-87ea-4173-8c91-1ee95524520e" (UID: "8d08feee-87ea-4173-8c91-1ee95524520e"). InnerVolumeSpecName "kube-api-access-nc9pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:36.371519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.371497    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca" pod "8d08feee-87ea-4173-8c91-1ee95524520e" (UID: "8d08feee-87ea-4173-8c91-1ee95524520e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:36.374673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.374641    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:38:36.374778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.374693    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:38:36.470442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.470414    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9pk\" (UniqueName: \"kubernetes.io/projected/8d08feee-87ea-4173-8c91-1ee95524520e-kube-api-access-nc9pk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:36.470442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:36.470445    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d08feee-87ea-4173-8c91-1ee95524520e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.141504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.141475    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"
Oct 30 21:38:37.141935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.141501    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg" event={"ID":"8d08feee-87ea-4173-8c91-1ee95524520e","Type":"ContainerDied","Data":"8bb2f7d1ac76b9135ae1d88100463c93c8d112498b14cbd03027b80b559ea04a"}
Oct 30 21:38:37.141935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.141554    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb2f7d1ac76b9135ae1d88100463c93c8d112498b14cbd03027b80b559ea04a"
Oct 30 21:38:37.143613 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.143575    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8" event={"ID":"b775af41-77b2-4daf-9b6b-c7097282db85","Type":"ContainerStarted","Data":"f445f8fac76e02b61fffb2b19fdee581a4da439f074bd4f2cef14ff14a3d302a"}
Oct 30 21:38:37.143724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.143640    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8" event={"ID":"b775af41-77b2-4daf-9b6b-c7097282db85","Type":"ContainerStarted","Data":"58f5af7a09425589796436bcb73e4e7f593cf4b13dadce19d79ade4ae1f05d29"}
Oct 30 21:38:37.145915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.145892    6695 generic.go:356] "Generic (PLEG): container finished" podID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerID="ca2546b0a76f691ffcdf89a9237b374b1d891bd6b9d012fd555a69a46df0474b" exitCode=0
Oct 30 21:38:37.146017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.145959    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" event={"ID":"f7c1f544-3c15-41fe-aed1-17449c0e7766","Type":"ContainerDied","Data":"ca2546b0a76f691ffcdf89a9237b374b1d891bd6b9d012fd555a69a46df0474b"}
Oct 30 21:38:37.146017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.146011    6695 scope.go:122] "RemoveContainer" containerID="f615c346ad4f1deba3c7580da28aba46133d6e7af342e49a070444bd36274b5c"
Oct 30 21:38:37.147805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.147783    6695 generic.go:356] "Generic (PLEG): container finished" podID="751c146a-33a9-4f3b-b924-3d6ad36e32ad" containerID="0dc4dd71a52710f9e0c018149ccdd3d15eb007398085314240b0b7332eabc55f" exitCode=0
Oct 30 21:38:37.147914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.147826    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d" event={"ID":"751c146a-33a9-4f3b-b924-3d6ad36e32ad","Type":"ContainerDied","Data":"0dc4dd71a52710f9e0c018149ccdd3d15eb007398085314240b0b7332eabc55f"}
Oct 30 21:38:37.253198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.253169    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:38:37.383162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383129    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383195    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383219    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383256    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383319    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.383561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383434    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383466    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383513    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383652    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy\") pod \"f7c1f544-3c15-41fe-aed1-17449c0e7766\" (UID: \"f7c1f544-3c15-41fe-aed1-17449c0e7766\") "
Oct 30 21:38:37.383923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383647    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.383923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383817    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.383923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.383868    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:37.384229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.384012    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.384229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.384170    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.384229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.384189    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1f544-3c15-41fe-aed1-17449c0e7766-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.384229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.384204    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.384229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.384217    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.384229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.384231    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.385441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.385419    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:37.385521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.385476    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.385521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.385499    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "kube-api-access-mds7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:37.385598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.385559    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token" pod "f7c1f544-3c15-41fe-aed1-17449c0e7766" (UID: "f7c1f544-3c15-41fe-aed1-17449c0e7766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:37.485589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.485563    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.485589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.485588    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.485746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.485597    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mds7s\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-kube-api-access-mds7s\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.485746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.485619    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c1f544-3c15-41fe-aed1-17449c0e7766-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.687839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.687812    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:38:37.789414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789378    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789446    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789469    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789522    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789631    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789687    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789732    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789796    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789849    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket\") pod \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\" (UID: \"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08\") "
Oct 30 21:38:37.789928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789872    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.790088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789958    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.790088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.789977    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:37.790213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790129    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.790213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790163    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.790410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790393    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.790410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790410    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.790542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790422    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.790542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790434    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.790542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.790443    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.791887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.791824    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "kube-api-access-9f5sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:37.792033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.792014    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:37.792127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.792100    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:38:37.792284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.792264    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy" pod "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" (UID: "8dbb82dc-8868-49ab-a6db-8a4fd4f62a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:37.891742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.891708    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.891742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.891735    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.891742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.891744    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:37.891977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:37.891753    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f5sl\" (UniqueName: \"kubernetes.io/projected/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08-kube-api-access-9f5sl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:38.155766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.155675    6695 generic.go:356] "Generic (PLEG): container finished" podID="b775af41-77b2-4daf-9b6b-c7097282db85" containerID="f445f8fac76e02b61fffb2b19fdee581a4da439f074bd4f2cef14ff14a3d302a" exitCode=0
Oct 30 21:38:38.156226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.155768    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8" event={"ID":"b775af41-77b2-4daf-9b6b-c7097282db85","Type":"ContainerDied","Data":"f445f8fac76e02b61fffb2b19fdee581a4da439f074bd4f2cef14ff14a3d302a"}
Oct 30 21:38:38.157435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.157415    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"
Oct 30 21:38:38.157566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.157426    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p" event={"ID":"f7c1f544-3c15-41fe-aed1-17449c0e7766","Type":"ContainerDied","Data":"de78553c7326d430b57735fb3c62bad70a76b272e2fa85c34d803f7318effbe6"}
Oct 30 21:38:38.157566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.157481    6695 scope.go:122] "RemoveContainer" containerID="ca2546b0a76f691ffcdf89a9237b374b1d891bd6b9d012fd555a69a46df0474b"
Oct 30 21:38:38.159630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.159580    6695 generic.go:356] "Generic (PLEG): container finished" podID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerID="06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075" exitCode=0
Oct 30 21:38:38.159760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.159627    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" event={"ID":"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08","Type":"ContainerDied","Data":"06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075"}
Oct 30 21:38:38.159846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.159794    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp" event={"ID":"8dbb82dc-8868-49ab-a6db-8a4fd4f62a08","Type":"ContainerDied","Data":"d55b1a744849820ba0f1f52f1e265375b2c1603901b1e44bd9f57f48ececd843"}
Oct 30 21:38:38.160158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.160140    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"
Oct 30 21:38:38.170516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.170494    6695 scope.go:122] "RemoveContainer" containerID="06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075"
Oct 30 21:38:38.183198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.183170    6695 scope.go:122] "RemoveContainer" containerID="81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"
Oct 30 21:38:38.197229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.196566    6695 scope.go:122] "RemoveContainer" containerID="06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075"
Oct 30 21:38:38.197913 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:38.197588    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075\": container with ID starting with 06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075 not found: ID does not exist" containerID="06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075"
Oct 30 21:38:38.198048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.197927    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075"} err="failed to get container status \"06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075\": rpc error: code = NotFound desc = could not find container \"06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075\": container with ID starting with 06e0947881a247093757244b53f0192aae883fc22bd431209f29dc15b083e075 not found: ID does not exist"
Oct 30 21:38:38.198048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.197960    6695 scope.go:122] "RemoveContainer" containerID="81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"
Oct 30 21:38:38.198365 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:38.198341    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb\": container with ID starting with 81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb not found: ID does not exist" containerID="81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"
Oct 30 21:38:38.198441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.198377    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb"} err="failed to get container status \"81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb\": rpc error: code = NotFound desc = could not find container \"81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb\": container with ID starting with 81c7a39002c337fbc5e33503e036b993f222870acc5da631fa9dc223dc520ffb not found: ID does not exist"
Oct 30 21:38:38.209665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.209588    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"]
Oct 30 21:38:38.220467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.220445    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-8a20c6-openshift-default-56669c4445-gzm4p"]
Oct 30 21:38:38.276224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.275918    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"]
Oct 30 21:38:38.277207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.277178    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1ee3-openshift-default-6645868b5c-blxlp"]
Oct 30 21:38:38.305068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.305038    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:38.330401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.330375    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:38.398011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.397977    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l\" (UniqueName: \"kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l\") pod \"b775af41-77b2-4daf-9b6b-c7097282db85\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") "
Oct 30 21:38:38.398011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398016    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca\") pod \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") "
Oct 30 21:38:38.398273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398140    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca\") pod \"b775af41-77b2-4daf-9b6b-c7097282db85\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") "
Oct 30 21:38:38.398273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398206    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca\") pod \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") "
Oct 30 21:38:38.398273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398270    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca\") pod \"b775af41-77b2-4daf-9b6b-c7097282db85\" (UID: \"b775af41-77b2-4daf-9b6b-c7097282db85\") "
Oct 30 21:38:38.398449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398326    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv\" (UniqueName: \"kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv\") pod \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\" (UID: \"751c146a-33a9-4f3b-b924-3d6ad36e32ad\") "
Oct 30 21:38:38.398549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398450    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca" pod "751c146a-33a9-4f3b-b924-3d6ad36e32ad" (UID: "751c146a-33a9-4f3b-b924-3d6ad36e32ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:38.398765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398637    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca" pod "b775af41-77b2-4daf-9b6b-c7097282db85" (UID: "b775af41-77b2-4daf-9b6b-c7097282db85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:38.398864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.398823    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:38.400503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.400478    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l" pod "b775af41-77b2-4daf-9b6b-c7097282db85" (UID: "b775af41-77b2-4daf-9b6b-c7097282db85"). InnerVolumeSpecName "kube-api-access-l579l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:38.400581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.400521    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv" pod "751c146a-33a9-4f3b-b924-3d6ad36e32ad" (UID: "751c146a-33a9-4f3b-b924-3d6ad36e32ad"). InnerVolumeSpecName "kube-api-access-f5xqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:38.400792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.400775    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca" pod "751c146a-33a9-4f3b-b924-3d6ad36e32ad" (UID: "751c146a-33a9-4f3b-b924-3d6ad36e32ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:38.401025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.401003    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca" pod "b775af41-77b2-4daf-9b6b-c7097282db85" (UID: "b775af41-77b2-4daf-9b6b-c7097282db85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:38.500030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.499999    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/751c146a-33a9-4f3b-b924-3d6ad36e32ad-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:38.500030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.500029    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:38.500182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.500040    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xqv\" (UniqueName: \"kubernetes.io/projected/751c146a-33a9-4f3b-b924-3d6ad36e32ad-kube-api-access-f5xqv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:38.500182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.500048    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l579l\" (UniqueName: \"kubernetes.io/projected/b775af41-77b2-4daf-9b6b-c7097282db85-kube-api-access-l579l\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:38.500182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:38.500059    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b775af41-77b2-4daf-9b6b-c7097282db85-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:39.170096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.170072    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"
Oct 30 21:38:39.170520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.170098    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d" event={"ID":"751c146a-33a9-4f3b-b924-3d6ad36e32ad","Type":"ContainerDied","Data":"dfc120f3042973f90a73bbcb043d95a1364063e0d493786375a0f39a9c5dcb5d"}
Oct 30 21:38:39.170520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.170182    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfc120f3042973f90a73bbcb043d95a1364063e0d493786375a0f39a9c5dcb5d"
Oct 30 21:38:39.171532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.171511    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"
Oct 30 21:38:39.171662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.171524    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8" event={"ID":"b775af41-77b2-4daf-9b6b-c7097282db85","Type":"ContainerDied","Data":"58f5af7a09425589796436bcb73e4e7f593cf4b13dadce19d79ade4ae1f05d29"}
Oct 30 21:38:39.171662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.171576    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f5af7a09425589796436bcb73e4e7f593cf4b13dadce19d79ade4ae1f05d29"
Oct 30 21:38:39.217333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.217299    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" path="/var/lib/kubelet/pods/8dbb82dc-8868-49ab-a6db-8a4fd4f62a08/volumes"
Oct 30 21:38:39.217912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:39.217885    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" path="/var/lib/kubelet/pods/f7c1f544-3c15-41fe-aed1-17449c0e7766/volumes"
Oct 30 21:38:44.374216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374181    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"]
Oct 30 21:38:44.374923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374903    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374925    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374947    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374954    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374980    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.374987    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375016    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b775af41-77b2-4daf-9b6b-c7097282db85" containerName="revoke-keys"
Oct 30 21:38:44.375031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375023    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b775af41-77b2-4daf-9b6b-c7097282db85" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375037    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375045    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375055    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8d08feee-87ea-4173-8c91-1ee95524520e" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375062    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d08feee-87ea-4173-8c91-1ee95524520e" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375078    6695 cpu_manager.go:414] "containerMap: removing container" podUID="751c146a-33a9-4f3b-b924-3d6ad36e32ad" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375085    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751c146a-33a9-4f3b-b924-3d6ad36e32ad" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375221    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375235    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d08feee-87ea-4173-8c91-1ee95524520e" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375245    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="751c146a-33a9-4f3b-b924-3d6ad36e32ad" containerName="revoke-keys"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375260    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375271    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbb82dc-8868-49ab-a6db-8a4fd4f62a08" containerName="istio-proxy"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375284    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c1f544-3c15-41fe-aed1-17449c0e7766" containerName="istio-proxy"
Oct 30 21:38:44.375373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.375297    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b775af41-77b2-4daf-9b6b-c7097282db85" containerName="revoke-keys"
Oct 30 21:38:44.379967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.379947    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.398879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.398706    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"]
Oct 30 21:38:44.406401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.405794    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:44.406401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.405921    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:44.406401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.406027    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:38:44.406401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.406093    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:44.481264 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.481235    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.481427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.481376    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.481515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.481475    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvv9\" (UniqueName: \"kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.605162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.596350    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.605162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.596482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvv9\" (UniqueName: \"kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.605162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.596619    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.605162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.597497    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.609434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.609401    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.618429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.618372    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvv9\" (UniqueName: \"kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9\") pod \"maas-api-revoke-sub-ccc0bb62-x8rs4\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.697868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.697779    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:44.995544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:44.995457    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"]
Oct 30 21:38:45.000659 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:45.000597    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb493d137_4288_4e93_900b_715f3771da7f.slice/crio-f3b3560ea7cc42809cdcae782ab6e13f3f7a4807471a185d40bc9ab414c5ebca WatchSource:0}: Error finding container f3b3560ea7cc42809cdcae782ab6e13f3f7a4807471a185d40bc9ab414c5ebca: Status 404 returned error can't find the container with id f3b3560ea7cc42809cdcae782ab6e13f3f7a4807471a185d40bc9ab414c5ebca
Oct 30 21:38:45.215159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.215119    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4" event={"ID":"b493d137-4288-4e93-900b-715f3771da7f","Type":"ContainerStarted","Data":"f3b3560ea7cc42809cdcae782ab6e13f3f7a4807471a185d40bc9ab414c5ebca"}
Oct 30 21:38:45.429273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.429235    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"]
Oct 30 21:38:45.433467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.433446    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.452094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.452071    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"]
Oct 30 21:38:45.508955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.508916    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.509139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.509072    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5f5\" (UniqueName: \"kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.509283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.509266    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.610026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.609937    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.610026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.610002    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.610274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.610101    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5f5\" (UniqueName: \"kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.610932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.610900    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.612576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.612550    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.629780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.629750    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5f5\" (UniqueName: \"kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5\") pod \"maas-api-revoke-sub-ef4b4acf-chzvf\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:45.747299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:45.747256    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:46.021283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:46.021247    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"]
Oct 30 21:38:46.052797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:46.052765    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"]
Oct 30 21:38:46.061720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:46.061690    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"]
Oct 30 21:38:46.082587 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:46.082558    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cbfa8f_350c_4a6a_9320_59c5959ae58a.slice/crio-d9071888906980ace320d05ac1cab0632aa241f0c2721091eb97f4b498d3420a WatchSource:0}: Error finding container d9071888906980ace320d05ac1cab0632aa241f0c2721091eb97f4b498d3420a: Status 404 returned error can't find the container with id d9071888906980ace320d05ac1cab0632aa241f0c2721091eb97f4b498d3420a
Oct 30 21:38:46.223507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:46.223472    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf" event={"ID":"91cbfa8f-350c-4a6a-9320-59c5959ae58a","Type":"ContainerStarted","Data":"d9071888906980ace320d05ac1cab0632aa241f0c2721091eb97f4b498d3420a"}
Oct 30 21:38:46.225229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:46.225202    6695 generic.go:356] "Generic (PLEG): container finished" podID="b493d137-4288-4e93-900b-715f3771da7f" containerID="21af11cffc85501dea2f0d41b8a1ccf4980607a227a44baa6abf93fdc51f9c5b" exitCode=0
Oct 30 21:38:46.225328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:46.225261    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4" event={"ID":"b493d137-4288-4e93-900b-715f3771da7f","Type":"ContainerDied","Data":"21af11cffc85501dea2f0d41b8a1ccf4980607a227a44baa6abf93fdc51f9c5b"}
Oct 30 21:38:47.232980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.232948    6695 generic.go:356] "Generic (PLEG): container finished" podID="91cbfa8f-350c-4a6a-9320-59c5959ae58a" containerID="bc7cf5c009ace93444af690d6dfb13d7e4c32ef96ef14d73f73c46b948f2c7f2" exitCode=0
Oct 30 21:38:47.233515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.233029    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf" event={"ID":"91cbfa8f-350c-4a6a-9320-59c5959ae58a","Type":"ContainerDied","Data":"bc7cf5c009ace93444af690d6dfb13d7e4c32ef96ef14d73f73c46b948f2c7f2"}
Oct 30 21:38:47.390633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.390594    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:47.532731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.532699    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca\") pod \"b493d137-4288-4e93-900b-715f3771da7f\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") "
Oct 30 21:38:47.532946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.532745    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca\") pod \"b493d137-4288-4e93-900b-715f3771da7f\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") "
Oct 30 21:38:47.532946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.532786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9\" (UniqueName: \"kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9\") pod \"b493d137-4288-4e93-900b-715f3771da7f\" (UID: \"b493d137-4288-4e93-900b-715f3771da7f\") "
Oct 30 21:38:47.533187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.533160    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca" pod "b493d137-4288-4e93-900b-715f3771da7f" (UID: "b493d137-4288-4e93-900b-715f3771da7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:47.534822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.534786    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca" pod "b493d137-4288-4e93-900b-715f3771da7f" (UID: "b493d137-4288-4e93-900b-715f3771da7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:47.534919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.534878    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9" pod "b493d137-4288-4e93-900b-715f3771da7f" (UID: "b493d137-4288-4e93-900b-715f3771da7f"). InnerVolumeSpecName "kube-api-access-vrvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:47.633862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.633824    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b493d137-4288-4e93-900b-715f3771da7f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:47.633862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.633858    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b493d137-4288-4e93-900b-715f3771da7f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:47.634052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:47.633874    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvv9\" (UniqueName: \"kubernetes.io/projected/b493d137-4288-4e93-900b-715f3771da7f-kube-api-access-vrvv9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:48.240674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.240641    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"
Oct 30 21:38:48.240674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.240661    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4" event={"ID":"b493d137-4288-4e93-900b-715f3771da7f","Type":"ContainerDied","Data":"f3b3560ea7cc42809cdcae782ab6e13f3f7a4807471a185d40bc9ab414c5ebca"}
Oct 30 21:38:48.241130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.240715    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b3560ea7cc42809cdcae782ab6e13f3f7a4807471a185d40bc9ab414c5ebca"
Oct 30 21:38:48.373878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.373843    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"]
Oct 30 21:38:48.374442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.374426    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b493d137-4288-4e93-900b-715f3771da7f" containerName="revoke-keys"
Oct 30 21:38:48.374490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.374443    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b493d137-4288-4e93-900b-715f3771da7f" containerName="revoke-keys"
Oct 30 21:38:48.374546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.374536    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b493d137-4288-4e93-900b-715f3771da7f" containerName="revoke-keys"
Oct 30 21:38:48.378465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.378440    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.397281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.397244    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"]
Oct 30 21:38:48.399808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.399783    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:48.442683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.442652    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca\") pod \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") "
Oct 30 21:38:48.442867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.442775    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5\" (UniqueName: \"kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5\") pod \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") "
Oct 30 21:38:48.442867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.442807    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca\") pod \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\" (UID: \"91cbfa8f-350c-4a6a-9320-59c5959ae58a\") "
Oct 30 21:38:48.443129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.443099    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cz8k\" (UniqueName: \"kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.443293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.443199    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca" pod "91cbfa8f-350c-4a6a-9320-59c5959ae58a" (UID: "91cbfa8f-350c-4a6a-9320-59c5959ae58a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:48.443368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.443288    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.443435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.443389    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.443549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.443531    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:48.444912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.444891    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca" pod "91cbfa8f-350c-4a6a-9320-59c5959ae58a" (UID: "91cbfa8f-350c-4a6a-9320-59c5959ae58a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:48.445023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.444916    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5" pod "91cbfa8f-350c-4a6a-9320-59c5959ae58a" (UID: "91cbfa8f-350c-4a6a-9320-59c5959ae58a"). InnerVolumeSpecName "kube-api-access-7p5f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:48.545163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.545118    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.545389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.545220    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cz8k\" (UniqueName: \"kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.545389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.545292    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.545389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.545344    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91cbfa8f-350c-4a6a-9320-59c5959ae58a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:48.545389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.545356    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5f5\" (UniqueName: \"kubernetes.io/projected/91cbfa8f-350c-4a6a-9320-59c5959ae58a-kube-api-access-7p5f5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:48.545829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.545810    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.547778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.547754    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.564985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.564949    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cz8k\" (UniqueName: \"kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k\") pod \"maas-api-revoke-sub-52d4454b-5fw5k\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:48.710829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:48.710785    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:49.018146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:49.018059    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"]
Oct 30 21:38:49.027085 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:49.027041    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe5d089_eb01_449a_a1fa_c9458937f92e.slice/crio-dff2a76cb15dac014a656ea07e9a65dee61b3720d420df11065bea1be3864241 WatchSource:0}: Error finding container dff2a76cb15dac014a656ea07e9a65dee61b3720d420df11065bea1be3864241: Status 404 returned error can't find the container with id dff2a76cb15dac014a656ea07e9a65dee61b3720d420df11065bea1be3864241
Oct 30 21:38:49.249302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:49.249260    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k" event={"ID":"2fe5d089-eb01-449a-a1fa-c9458937f92e","Type":"ContainerStarted","Data":"dff2a76cb15dac014a656ea07e9a65dee61b3720d420df11065bea1be3864241"}
Oct 30 21:38:49.250953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:49.250925    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"
Oct 30 21:38:49.251108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:49.250952    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf" event={"ID":"91cbfa8f-350c-4a6a-9320-59c5959ae58a","Type":"ContainerDied","Data":"d9071888906980ace320d05ac1cab0632aa241f0c2721091eb97f4b498d3420a"}
Oct 30 21:38:49.251108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:49.250997    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9071888906980ace320d05ac1cab0632aa241f0c2721091eb97f4b498d3420a"
Oct 30 21:38:50.261668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:50.261526    6695 generic.go:356] "Generic (PLEG): container finished" podID="2fe5d089-eb01-449a-a1fa-c9458937f92e" containerID="3233520bcf0d4092930cc3c18f186178627fe6ebea747da4543767ec4d461c15" exitCode=0
Oct 30 21:38:50.261668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:50.261637    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k" event={"ID":"2fe5d089-eb01-449a-a1fa-c9458937f92e","Type":"ContainerDied","Data":"3233520bcf0d4092930cc3c18f186178627fe6ebea747da4543767ec4d461c15"}
Oct 30 21:38:50.769581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:50.769540    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"]
Oct 30 21:38:50.769834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:50.769797    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs" podUID="fccddbf8-0957-46b5-b911-d9f80264887a" containerName="payload-processing" containerID="cri-o://77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087" gracePeriod=30
Oct 30 21:38:50.786650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:50.786585    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"]
Oct 30 21:38:50.786901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:50.786858    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62" podUID="03ac03f9-4225-43cd-bed1-456553318b9f" containerName="payload-pre-processing" containerID="cri-o://a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf" gracePeriod=30
Oct 30 21:38:51.022291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.022227    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l" podUID="264a8f62-181f-4cdf-b5d1-359cb9c52925" containerName="payload-processing" containerID="cri-o://9fd69e4de920ca44810fc9ba0175210aa009fc4f8f28457665f78d13d284cb1a" gracePeriod=25
Oct 30 21:38:51.055031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.053198    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd" podUID="b1fa6b5e-cad0-43e0-a034-12061c05b741" containerName="payload-pre-processing" containerID="cri-o://32d80a88c291c0ea790335999ebc25f953fcdcd39ca28307b8a0893f0168b766" gracePeriod=25
Oct 30 21:38:51.212471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.212436    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:38:51.216622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.216562    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:38:51.276030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.275932    6695 generic.go:356] "Generic (PLEG): container finished" podID="03ac03f9-4225-43cd-bed1-456553318b9f" containerID="a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf" exitCode=0
Oct 30 21:38:51.276500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.276059    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62" event={"ID":"03ac03f9-4225-43cd-bed1-456553318b9f","Type":"ContainerDied","Data":"a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf"}
Oct 30 21:38:51.276500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.276113    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62" event={"ID":"03ac03f9-4225-43cd-bed1-456553318b9f","Type":"ContainerDied","Data":"f93eb3e0d56b6ac0bc9c0204f472d8de08cc84ca1905bcd43a074581a537a84a"}
Oct 30 21:38:51.276500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.276146    6695 scope.go:122] "RemoveContainer" containerID="a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf"
Oct 30 21:38:51.276500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.276324    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"
Oct 30 21:38:51.278539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.278507    6695 generic.go:356] "Generic (PLEG): container finished" podID="b1fa6b5e-cad0-43e0-a034-12061c05b741" containerID="32d80a88c291c0ea790335999ebc25f953fcdcd39ca28307b8a0893f0168b766" exitCode=0
Oct 30 21:38:51.278793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.278686    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd" event={"ID":"b1fa6b5e-cad0-43e0-a034-12061c05b741","Type":"ContainerDied","Data":"32d80a88c291c0ea790335999ebc25f953fcdcd39ca28307b8a0893f0168b766"}
Oct 30 21:38:51.280359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.280328    6695 generic.go:356] "Generic (PLEG): container finished" podID="fccddbf8-0957-46b5-b911-d9f80264887a" containerID="77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087" exitCode=0
Oct 30 21:38:51.280487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.280471    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs" event={"ID":"fccddbf8-0957-46b5-b911-d9f80264887a","Type":"ContainerDied","Data":"77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087"}
Oct 30 21:38:51.280559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.280525    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs" event={"ID":"fccddbf8-0957-46b5-b911-d9f80264887a","Type":"ContainerDied","Data":"7bd757526ed489f96a4ba7ed4cdec0ac3d41c3203f0a62571c0e5bd42bf2f1f8"}
Oct 30 21:38:51.280787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.280628    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"
Oct 30 21:38:51.283164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.282945    6695 generic.go:356] "Generic (PLEG): container finished" podID="264a8f62-181f-4cdf-b5d1-359cb9c52925" containerID="9fd69e4de920ca44810fc9ba0175210aa009fc4f8f28457665f78d13d284cb1a" exitCode=0
Oct 30 21:38:51.283290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.283184    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume\") pod \"03ac03f9-4225-43cd-bed1-456553318b9f\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") "
Oct 30 21:38:51.283290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.283239    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk\" (UniqueName: \"kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk\") pod \"03ac03f9-4225-43cd-bed1-456553318b9f\" (UID: \"03ac03f9-4225-43cd-bed1-456553318b9f\") "
Oct 30 21:38:51.283365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.283315    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs\" (UniqueName: \"kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs\") pod \"fccddbf8-0957-46b5-b911-d9f80264887a\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") "
Oct 30 21:38:51.283519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.283497    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume\") pod \"fccddbf8-0957-46b5-b911-d9f80264887a\" (UID: \"fccddbf8-0957-46b5-b911-d9f80264887a\") "
Oct 30 21:38:51.283656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.283567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l" event={"ID":"264a8f62-181f-4cdf-b5d1-359cb9c52925","Type":"ContainerDied","Data":"9fd69e4de920ca44810fc9ba0175210aa009fc4f8f28457665f78d13d284cb1a"}
Oct 30 21:38:51.284274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.284244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume" pod "03ac03f9-4225-43cd-bed1-456553318b9f" (UID: "03ac03f9-4225-43cd-bed1-456553318b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:51.284588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.284567    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume" pod "fccddbf8-0957-46b5-b911-d9f80264887a" (UID: "fccddbf8-0957-46b5-b911-d9f80264887a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:51.295700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.290459    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk" pod "03ac03f9-4225-43cd-bed1-456553318b9f" (UID: "03ac03f9-4225-43cd-bed1-456553318b9f"). InnerVolumeSpecName "kube-api-access-ql8tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.297377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.296490    6695 scope.go:122] "RemoveContainer" containerID="a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf"
Oct 30 21:38:51.297377 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:51.296926    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf\": container with ID starting with a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf not found: ID does not exist" containerID="a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf"
Oct 30 21:38:51.297377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.296973    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf"} err="failed to get container status \"a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf\": rpc error: code = NotFound desc = could not find container \"a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf\": container with ID starting with a1bb44d3bd3e1b32b80ec8866339e42bdcac806dbbc819a59f5ab977eb8a1edf not found: ID does not exist"
Oct 30 21:38:51.297377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.296991    6695 scope.go:122] "RemoveContainer" containerID="77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087"
Oct 30 21:38:51.316301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.316252    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs" pod "fccddbf8-0957-46b5-b911-d9f80264887a" (UID: "fccddbf8-0957-46b5-b911-d9f80264887a"). InnerVolumeSpecName "kube-api-access-l5ccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.323269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.323199    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"]
Oct 30 21:38:51.326906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.326867    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0e991-b5vzg"]
Oct 30 21:38:51.333394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.333363    6695 scope.go:122] "RemoveContainer" containerID="77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087"
Oct 30 21:38:51.336838 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:38:51.336755    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087\": container with ID starting with 77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087 not found: ID does not exist" containerID="77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087"
Oct 30 21:38:51.336838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.336806    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087"} err="failed to get container status \"77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087\": rpc error: code = NotFound desc = could not find container \"77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087\": container with ID starting with 77d0a5ad6b4bd8f44c23ce2742381b013f8626c51683299123870a2a2680b087 not found: ID does not exist"
Oct 30 21:38:51.346990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.346956    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:38:51.385647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.385574    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fccddbf8-0957-46b5-b911-d9f80264887a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.385858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.385840    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03ac03f9-4225-43cd-bed1-456553318b9f-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.385987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.385973    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8tk\" (UniqueName: \"kubernetes.io/projected/03ac03f9-4225-43cd-bed1-456553318b9f-kube-api-access-ql8tk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.386081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.386070    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ccs\" (UniqueName: \"kubernetes.io/projected/fccddbf8-0957-46b5-b911-d9f80264887a-kube-api-access-l5ccs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.393960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.393921    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:38:51.508661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.507031    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt\" (UniqueName: \"kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt\") pod \"264a8f62-181f-4cdf-b5d1-359cb9c52925\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") "
Oct 30 21:38:51.508661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.507100    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume\") pod \"264a8f62-181f-4cdf-b5d1-359cb9c52925\" (UID: \"264a8f62-181f-4cdf-b5d1-359cb9c52925\") "
Oct 30 21:38:51.508661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.507274    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf\" (UniqueName: \"kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf\") pod \"b1fa6b5e-cad0-43e0-a034-12061c05b741\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") "
Oct 30 21:38:51.508661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.507370    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume\") pod \"b1fa6b5e-cad0-43e0-a034-12061c05b741\" (UID: \"b1fa6b5e-cad0-43e0-a034-12061c05b741\") "
Oct 30 21:38:51.508661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.508324    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume" pod "b1fa6b5e-cad0-43e0-a034-12061c05b741" (UID: "b1fa6b5e-cad0-43e0-a034-12061c05b741"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:51.528704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.527898    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume" pod "264a8f62-181f-4cdf-b5d1-359cb9c52925" (UID: "264a8f62-181f-4cdf-b5d1-359cb9c52925"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:51.566024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.565961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf" pod "b1fa6b5e-cad0-43e0-a034-12061c05b741" (UID: "b1fa6b5e-cad0-43e0-a034-12061c05b741"). InnerVolumeSpecName "kube-api-access-7lmhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.566818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.566784    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt" pod "264a8f62-181f-4cdf-b5d1-359cb9c52925" (UID: "264a8f62-181f-4cdf-b5d1-359cb9c52925"). InnerVolumeSpecName "kube-api-access-mnrxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.595934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.595901    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:51.609592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.609555    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lmhf\" (UniqueName: \"kubernetes.io/projected/b1fa6b5e-cad0-43e0-a034-12061c05b741-kube-api-access-7lmhf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.609952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.609929    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1fa6b5e-cad0-43e0-a034-12061c05b741-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.610640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.610617    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnrxt\" (UniqueName: \"kubernetes.io/projected/264a8f62-181f-4cdf-b5d1-359cb9c52925-kube-api-access-mnrxt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.610799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.610785    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264a8f62-181f-4cdf-b5d1-359cb9c52925-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.672002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.671963    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"]
Oct 30 21:38:51.676220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.676187    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b0e991-5f6ddc6dcb-svrfs"]
Oct 30 21:38:51.708620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.708560    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"]
Oct 30 21:38:51.712112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.712070    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca\") pod \"2fe5d089-eb01-449a-a1fa-c9458937f92e\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") "
Oct 30 21:38:51.712365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.712074    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-568d98bb59-f4r62"]
Oct 30 21:38:51.712365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.712265    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca\") pod \"2fe5d089-eb01-449a-a1fa-c9458937f92e\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") "
Oct 30 21:38:51.712365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.712294    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k\" (UniqueName: \"kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k\") pod \"2fe5d089-eb01-449a-a1fa-c9458937f92e\" (UID: \"2fe5d089-eb01-449a-a1fa-c9458937f92e\") "
Oct 30 21:38:51.712706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.712680    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca" pod "2fe5d089-eb01-449a-a1fa-c9458937f92e" (UID: "2fe5d089-eb01-449a-a1fa-c9458937f92e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:51.712826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.712808    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.714360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.714325    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca" pod "2fe5d089-eb01-449a-a1fa-c9458937f92e" (UID: "2fe5d089-eb01-449a-a1fa-c9458937f92e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:51.714523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.714502    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k" pod "2fe5d089-eb01-449a-a1fa-c9458937f92e" (UID: "2fe5d089-eb01-449a-a1fa-c9458937f92e"). InnerVolumeSpecName "kube-api-access-6cz8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.753593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.753557    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"]
Oct 30 21:38:51.756559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.756520    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-912e61-5cwv8"]
Oct 30 21:38:51.814516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.814404    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe5d089-eb01-449a-a1fa-c9458937f92e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.814516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:51.814434    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cz8k\" (UniqueName: \"kubernetes.io/projected/2fe5d089-eb01-449a-a1fa-c9458937f92e-kube-api-access-6cz8k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:52.292853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.292822    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"
Oct 30 21:38:52.293376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.292854    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd" event={"ID":"b1fa6b5e-cad0-43e0-a034-12061c05b741","Type":"ContainerDied","Data":"9427a82325b9a68312d12e99ef68d2bc72362270aaaca5734bea4cb15056515e"}
Oct 30 21:38:52.293376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.292937    6695 scope.go:122] "RemoveContainer" containerID="32d80a88c291c0ea790335999ebc25f953fcdcd39ca28307b8a0893f0168b766"
Oct 30 21:38:52.295720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.295694    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"
Oct 30 21:38:52.295862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.295724    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l" event={"ID":"264a8f62-181f-4cdf-b5d1-359cb9c52925","Type":"ContainerDied","Data":"2cf6fce02b24899a5dae011b1f8dc1889d48b6ad0663851edd37d8fdfed4168f"}
Oct 30 21:38:52.297569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.297545    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"
Oct 30 21:38:52.297767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.297572    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k" event={"ID":"2fe5d089-eb01-449a-a1fa-c9458937f92e","Type":"ContainerDied","Data":"dff2a76cb15dac014a656ea07e9a65dee61b3720d420df11065bea1be3864241"}
Oct 30 21:38:52.297767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.297637    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dff2a76cb15dac014a656ea07e9a65dee61b3720d420df11065bea1be3864241"
Oct 30 21:38:52.306716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.306691    6695 scope.go:122] "RemoveContainer" containerID="9fd69e4de920ca44810fc9ba0175210aa009fc4f8f28457665f78d13d284cb1a"
Oct 30 21:38:52.362652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.359098    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"]
Oct 30 21:38:52.369703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.369646    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-912e61-558788ff7f-z7jbd"]
Oct 30 21:38:52.414628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.414371    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"]
Oct 30 21:38:52.415012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:52.414982    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-912e61-7b5c8f57d7-gn94l"]
Oct 30 21:38:53.248139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:53.242245    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ac03f9-4225-43cd-bed1-456553318b9f" path="/var/lib/kubelet/pods/03ac03f9-4225-43cd-bed1-456553318b9f/volumes"
Oct 30 21:38:53.248139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:53.243076    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264a8f62-181f-4cdf-b5d1-359cb9c52925" path="/var/lib/kubelet/pods/264a8f62-181f-4cdf-b5d1-359cb9c52925/volumes"
Oct 30 21:38:53.248139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:53.243874    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d08feee-87ea-4173-8c91-1ee95524520e" path="/var/lib/kubelet/pods/8d08feee-87ea-4173-8c91-1ee95524520e/volumes"
Oct 30 21:38:53.248139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:53.244999    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1fa6b5e-cad0-43e0-a034-12061c05b741" path="/var/lib/kubelet/pods/b1fa6b5e-cad0-43e0-a034-12061c05b741/volumes"
Oct 30 21:38:53.248139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:53.245718    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b775af41-77b2-4daf-9b6b-c7097282db85" path="/var/lib/kubelet/pods/b775af41-77b2-4daf-9b6b-c7097282db85/volumes"
Oct 30 21:38:53.248139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:53.246364    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fccddbf8-0957-46b5-b911-d9f80264887a" path="/var/lib/kubelet/pods/fccddbf8-0957-46b5-b911-d9f80264887a/volumes"
Oct 30 21:38:54.710859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.710819    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"]
Oct 30 21:38:54.711331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711320    6695 cpu_manager.go:414] "containerMap: removing container" podUID="91cbfa8f-350c-4a6a-9320-59c5959ae58a" containerName="revoke-keys"
Oct 30 21:38:54.711388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711333    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cbfa8f-350c-4a6a-9320-59c5959ae58a" containerName="revoke-keys"
Oct 30 21:38:54.711388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711356    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2fe5d089-eb01-449a-a1fa-c9458937f92e" containerName="revoke-keys"
Oct 30 21:38:54.711388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711362    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe5d089-eb01-449a-a1fa-c9458937f92e" containerName="revoke-keys"
Oct 30 21:38:54.711388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711372    6695 cpu_manager.go:414] "containerMap: removing container" podUID="03ac03f9-4225-43cd-bed1-456553318b9f" containerName="payload-pre-processing"
Oct 30 21:38:54.711388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711379    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ac03f9-4225-43cd-bed1-456553318b9f" containerName="payload-pre-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711394    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b1fa6b5e-cad0-43e0-a034-12061c05b741" containerName="payload-pre-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711399    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fa6b5e-cad0-43e0-a034-12061c05b741" containerName="payload-pre-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711411    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fccddbf8-0957-46b5-b911-d9f80264887a" containerName="payload-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711416    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fccddbf8-0957-46b5-b911-d9f80264887a" containerName="payload-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711434    6695 cpu_manager.go:414] "containerMap: removing container" podUID="264a8f62-181f-4cdf-b5d1-359cb9c52925" containerName="payload-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711439    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264a8f62-181f-4cdf-b5d1-359cb9c52925" containerName="payload-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711507    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe5d089-eb01-449a-a1fa-c9458937f92e" containerName="revoke-keys"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711516    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ac03f9-4225-43cd-bed1-456553318b9f" containerName="payload-pre-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711524    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fa6b5e-cad0-43e0-a034-12061c05b741" containerName="payload-pre-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711531    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fccddbf8-0957-46b5-b911-d9f80264887a" containerName="payload-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711539    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="264a8f62-181f-4cdf-b5d1-359cb9c52925" containerName="payload-processing"
Oct 30 21:38:54.711553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.711548    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cbfa8f-350c-4a6a-9320-59c5959ae58a" containerName="revoke-keys"
Oct 30 21:38:54.716914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.716876    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.742708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.742673    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:54.742948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.742674    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:54.743029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.742717    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:54.743029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.742819    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:38:54.745517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.745487    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"]
Oct 30 21:38:54.795493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.795452    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xjb\" (UniqueName: \"kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.795729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.795506    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.795729 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.795579    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.896875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.896828    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xjb\" (UniqueName: \"kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.896875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.896886    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.897165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.896914    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.897721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.897687    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.899393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.899373    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:54.915682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:54.915641    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xjb\" (UniqueName: \"kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb\") pod \"maas-api-revoke-sub-ab3dedce-n5npc\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:55.031832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:55.031792    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:55.335381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:55.335299    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"]
Oct 30 21:38:55.339663 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:55.339595    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2468efc_f5d8_4900_8d14_7565e567f72f.slice/crio-c715d4ac50a0cdd1541c7871d08ac791736834007404df6ddfc6def113ce1f55 WatchSource:0}: Error finding container c715d4ac50a0cdd1541c7871d08ac791736834007404df6ddfc6def113ce1f55: Status 404 returned error can't find the container with id c715d4ac50a0cdd1541c7871d08ac791736834007404df6ddfc6def113ce1f55
Oct 30 21:38:56.335828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:56.335794    6695 generic.go:356] "Generic (PLEG): container finished" podID="c2468efc-f5d8-4900-8d14-7565e567f72f" containerID="4b9251859083d9fb57bf3648112e38976227dc33f545cdcf82f438fe572c9a61" exitCode=0
Oct 30 21:38:56.336208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:56.335872    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc" event={"ID":"c2468efc-f5d8-4900-8d14-7565e567f72f","Type":"ContainerDied","Data":"4b9251859083d9fb57bf3648112e38976227dc33f545cdcf82f438fe572c9a61"}
Oct 30 21:38:56.336208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:56.335923    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc" event={"ID":"c2468efc-f5d8-4900-8d14-7565e567f72f","Type":"ContainerStarted","Data":"c715d4ac50a0cdd1541c7871d08ac791736834007404df6ddfc6def113ce1f55"}
Oct 30 21:38:57.500312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.500280    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:57.529548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.529511    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca\") pod \"c2468efc-f5d8-4900-8d14-7565e567f72f\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") "
Oct 30 21:38:57.529786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.529584    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb\" (UniqueName: \"kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb\") pod \"c2468efc-f5d8-4900-8d14-7565e567f72f\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") "
Oct 30 21:38:57.529786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.529698    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca\") pod \"c2468efc-f5d8-4900-8d14-7565e567f72f\" (UID: \"c2468efc-f5d8-4900-8d14-7565e567f72f\") "
Oct 30 21:38:57.530034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.530004    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca" pod "c2468efc-f5d8-4900-8d14-7565e567f72f" (UID: "c2468efc-f5d8-4900-8d14-7565e567f72f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:57.530110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.530090    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:57.531939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.531896    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb" pod "c2468efc-f5d8-4900-8d14-7565e567f72f" (UID: "c2468efc-f5d8-4900-8d14-7565e567f72f"). InnerVolumeSpecName "kube-api-access-z8xjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:57.532044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.531973    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca" pod "c2468efc-f5d8-4900-8d14-7565e567f72f" (UID: "c2468efc-f5d8-4900-8d14-7565e567f72f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:57.631021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.630921    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2468efc-f5d8-4900-8d14-7565e567f72f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:57.631021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.630958    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xjb\" (UniqueName: \"kubernetes.io/projected/c2468efc-f5d8-4900-8d14-7565e567f72f-kube-api-access-z8xjb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:38:57.672214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.672170    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"]
Oct 30 21:38:57.672836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.672812    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c2468efc-f5d8-4900-8d14-7565e567f72f" containerName="revoke-keys"
Oct 30 21:38:57.672836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.672835    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2468efc-f5d8-4900-8d14-7565e567f72f" containerName="revoke-keys"
Oct 30 21:38:57.673048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.672963    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2468efc-f5d8-4900-8d14-7565e567f72f" containerName="revoke-keys"
Oct 30 21:38:57.676439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.676415    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.720130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.720095    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"]
Oct 30 21:38:57.732241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732203    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732268    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732325    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732366    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thjzg\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732398    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732499    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732617    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732805    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.733003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.732839    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834131    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834185    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834260    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834308    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834361    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834401    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thjzg\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834435    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834474    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834499    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834847    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.834949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.834890    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.835177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.835149    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.835318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.835211    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.835318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.835266    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.836751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.836730    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.836978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.836960    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.852508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.852465    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.852742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.852542    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thjzg\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:57.993789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:57.993692    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:58.302214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.302172    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"]
Oct 30 21:38:58.307488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.307451    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.326907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.326873    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"]
Oct 30 21:38:58.354090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.354059    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"
Oct 30 21:38:58.354273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.354094    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc" event={"ID":"c2468efc-f5d8-4900-8d14-7565e567f72f","Type":"ContainerDied","Data":"c715d4ac50a0cdd1541c7871d08ac791736834007404df6ddfc6def113ce1f55"}
Oct 30 21:38:58.354273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.354149    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c715d4ac50a0cdd1541c7871d08ac791736834007404df6ddfc6def113ce1f55"
Oct 30 21:38:58.358290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.358261    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.358462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.358333    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.358636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.358576    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvkm\" (UniqueName: \"kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.361098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.361069    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"]
Oct 30 21:38:58.363415 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:38:58.363383    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fbc84f_5818_40c6_ae6a_04c205d108ee.slice/crio-06b56e7a2ed1e6fd8a2c1f2627d4fd05ee447a691a60bcb24c729d7e60ab1f84 WatchSource:0}: Error finding container 06b56e7a2ed1e6fd8a2c1f2627d4fd05ee447a691a60bcb24c729d7e60ab1f84: Status 404 returned error can't find the container with id 06b56e7a2ed1e6fd8a2c1f2627d4fd05ee447a691a60bcb24c729d7e60ab1f84
Oct 30 21:38:58.366537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.366503    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.366627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.366575    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.366685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.366671    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.459378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.459342    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvkm\" (UniqueName: \"kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.459575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.459414    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.459575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.459480    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.460219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.460184    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.461948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.461922    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.505473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.505427    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvkm\" (UniqueName: \"kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm\") pod \"maas-api-revoke-sub-fea206f8-9q882\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:58.555351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.555243    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"]
Oct 30 21:38:58.555568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.555542    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy" containerID="cri-o://2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08" gracePeriod=30
Oct 30 21:38:58.624230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:58.624186    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:38:59.042630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.040854    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"]
Oct 30 21:38:59.363540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.363498    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" event={"ID":"98fbc84f-5818-40c6-ae6a-04c205d108ee","Type":"ContainerStarted","Data":"cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d"}
Oct 30 21:38:59.363767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.363568    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" event={"ID":"98fbc84f-5818-40c6-ae6a-04c205d108ee","Type":"ContainerStarted","Data":"06b56e7a2ed1e6fd8a2c1f2627d4fd05ee447a691a60bcb24c729d7e60ab1f84"}
Oct 30 21:38:59.365056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.365025    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882" event={"ID":"b5068dcf-7769-4224-81aa-a1b2189001a8","Type":"ContainerStarted","Data":"f8129cec5f7a3bfbdd9c326e564dca797b7eaa71dff3473a73f16724651765c5"}
Oct 30 21:38:59.443947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.443891    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podStartSLOduration=2.4438763740000002 podStartE2EDuration="2.443876374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.443503835 +0000 UTC m=+1722.365962133" watchObservedRunningTime="2026-10-30 21:38:59.443876374 +0000 UTC m=+1722.366334662"
Oct 30 21:38:59.562324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.562288    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"]
Oct 30 21:38:59.563075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.563046    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" containerID="cri-o://5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0" gracePeriod=30
Oct 30 21:38:59.994246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.994199    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:38:59.996041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.996010    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:38:59.996186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:38:59.996082    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 21:39:00.374351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.374313    6695 generic.go:356] "Generic (PLEG): container finished" podID="b5068dcf-7769-4224-81aa-a1b2189001a8" containerID="871cf2815948eab3fbc9025257e7371e407fd8edf3dde80293af5cbeceb62ec8" exitCode=0
Oct 30 21:39:00.374519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.374388    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882" event={"ID":"b5068dcf-7769-4224-81aa-a1b2189001a8","Type":"ContainerDied","Data":"871cf2815948eab3fbc9025257e7371e407fd8edf3dde80293af5cbeceb62ec8"}
Oct 30 21:39:00.592960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.592923    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"]
Oct 30 21:39:00.597856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.597829    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.645857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.645817    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c19b75-openshift-default-dockercfg-ww26j\""
Oct 30 21:39:00.649451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.649417    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"]
Oct 30 21:39:00.691091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691053    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691108    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691205    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcdt\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691303    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691334    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691368    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691545    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.691664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.691647    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.692155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.692135    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.803250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.797824    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.803250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.797926    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.803250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.798028    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.803250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.798120    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.803250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.798155    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcdt\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.803250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.798782    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.808623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.805570    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.808623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.805761    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.808623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.805807    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.808623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.805869    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.808623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.805930    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.830635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.823735    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.830635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.823995    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.830635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.825030    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.838656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.838201    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.868677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.854437    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcdt\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.868677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.861016    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.868677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.861699    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token\") pod \"e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") " pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.951352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.951187    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:00.994656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.994599    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:39:00.994840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:00.994687    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 21:39:01.137096 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:39:01.136857    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:39:01.353426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.353384    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"]
Oct 30 21:39:01.357135 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:01.357101    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10ff2df_7e2d_4273_b248_250f6f73d33e.slice/crio-80ef3f16f69003111480e1de11923a83a21afddff2ec164cfe958f6ea65c5f05 WatchSource:0}: Error finding container 80ef3f16f69003111480e1de11923a83a21afddff2ec164cfe958f6ea65c5f05: Status 404 returned error can't find the container with id 80ef3f16f69003111480e1de11923a83a21afddff2ec164cfe958f6ea65c5f05
Oct 30 21:39:01.359777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.359739    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:01.359860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.359817    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:01.359860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.359846    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:01.385308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.385282    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" event={"ID":"f10ff2df-7e2d-4273-b248-250f6f73d33e","Type":"ContainerStarted","Data":"80ef3f16f69003111480e1de11923a83a21afddff2ec164cfe958f6ea65c5f05"}
Oct 30 21:39:01.565737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.565581    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:39:01.658456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.658356    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca\") pod \"b5068dcf-7769-4224-81aa-a1b2189001a8\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") "
Oct 30 21:39:01.658997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.658511    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm\" (UniqueName: \"kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm\") pod \"b5068dcf-7769-4224-81aa-a1b2189001a8\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") "
Oct 30 21:39:01.658997 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.658553    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca\") pod \"b5068dcf-7769-4224-81aa-a1b2189001a8\" (UID: \"b5068dcf-7769-4224-81aa-a1b2189001a8\") "
Oct 30 21:39:01.659108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.659018    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca" pod "b5068dcf-7769-4224-81aa-a1b2189001a8" (UID: "b5068dcf-7769-4224-81aa-a1b2189001a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:01.661029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.660985    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm" pod "b5068dcf-7769-4224-81aa-a1b2189001a8" (UID: "b5068dcf-7769-4224-81aa-a1b2189001a8"). InnerVolumeSpecName "kube-api-access-7qvkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:01.663344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.663303    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca" pod "b5068dcf-7769-4224-81aa-a1b2189001a8" (UID: "b5068dcf-7769-4224-81aa-a1b2189001a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:01.724654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.720259    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:39:01.724654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.720324    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:39:01.760410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.760342    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvkm\" (UniqueName: \"kubernetes.io/projected/b5068dcf-7769-4224-81aa-a1b2189001a8-kube-api-access-7qvkm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:01.760410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.760376    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:01.760410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.760387    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5068dcf-7769-4224-81aa-a1b2189001a8-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:01.994297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.994209    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:39:01.994297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:01.994267    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 21:39:02.392573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.392539    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"
Oct 30 21:39:02.392757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.392570    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882" event={"ID":"b5068dcf-7769-4224-81aa-a1b2189001a8","Type":"ContainerDied","Data":"f8129cec5f7a3bfbdd9c326e564dca797b7eaa71dff3473a73f16724651765c5"}
Oct 30 21:39:02.392757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.392638    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8129cec5f7a3bfbdd9c326e564dca797b7eaa71dff3473a73f16724651765c5"
Oct 30 21:39:02.394562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.394532    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" event={"ID":"f10ff2df-7e2d-4273-b248-250f6f73d33e","Type":"ContainerStarted","Data":"cce57ec403c035183799d0b9483f0a91143fbe077056811bedd8f2ee99e3090d"}
Oct 30 21:39:02.634646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.630968    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podStartSLOduration=2.630946191 podStartE2EDuration="2.630946191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:02.459835474 +0000 UTC m=+1725.382293773" watchObservedRunningTime="2026-10-30 21:39:02.630946191 +0000 UTC m=+1725.553404491"
Oct 30 21:39:02.951923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.951741    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:02.957947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.957914    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:02.994977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.994943    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:39:02.995181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:02.994996    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 21:39:03.402635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.402586    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:03.403882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.403850    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:03.904395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.904365    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:39:03.992062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.991576    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.992657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.991979    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:03.992880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.992852    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.993085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.993066    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.993533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.993416    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:03.993690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.993486    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.993898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.993878    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.994038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994022    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.994165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994150    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.994314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994298    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.994538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994519    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token\") pod \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\" (UID: \"7c1730f8-e0e3-4345-9952-cde0a41a63b5\") "
Oct 30 21:39:03.994772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994209    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:39:03.994868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994836    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 21:39:03.994985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994435    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:03.994985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994617    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:03.994985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.994754    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:03.995720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.995698    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:03.995858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.995841    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:03.995955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.995941    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:03.996099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.996085    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:03.996213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.996199    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:03.996308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.996083    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "kube-api-access-ftwhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:03.996949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.996926    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:03.997049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.996990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:03.997154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:03.997133    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo" pod "7c1730f8-e0e3-4345-9952-cde0a41a63b5" (UID: "7c1730f8-e0e3-4345-9952-cde0a41a63b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:39:04.097677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.097622    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:04.097677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.097670    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftwhk\" (UniqueName: \"kubernetes.io/projected/7c1730f8-e0e3-4345-9952-cde0a41a63b5-kube-api-access-ftwhk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:04.097677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.097693    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:04.097942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.097704    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1730f8-e0e3-4345-9952-cde0a41a63b5-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:04.436696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.434940    6695 generic.go:356] "Generic (PLEG): container finished" podID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerID="2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08" exitCode=0
Oct 30 21:39:04.436696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.435982    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"
Oct 30 21:39:04.436998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.436924    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" event={"ID":"7c1730f8-e0e3-4345-9952-cde0a41a63b5","Type":"ContainerDied","Data":"2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08"}
Oct 30 21:39:04.436998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.436987    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4" event={"ID":"7c1730f8-e0e3-4345-9952-cde0a41a63b5","Type":"ContainerDied","Data":"c68b394f5b56b022eb8ac8fbe9cfdbe37df826f981d46fad05b1f8cbb218d13e"}
Oct 30 21:39:04.437152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.437018    6695 scope.go:122] "RemoveContainer" containerID="2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08"
Oct 30 21:39:04.449511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.449488    6695 scope.go:122] "RemoveContainer" containerID="8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"
Oct 30 21:39:04.460589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.460563    6695 scope.go:122] "RemoveContainer" containerID="2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08"
Oct 30 21:39:04.460967 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:39:04.460917    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08\": container with ID starting with 2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08 not found: ID does not exist" containerID="2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08"
Oct 30 21:39:04.461047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.460983    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08"} err="failed to get container status \"2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08\": rpc error: code = NotFound desc = could not find container \"2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08\": container with ID starting with 2cdf29bc85319cc3e03a0e947ad47d583fa4d42d07c0d9d2def4fe0c4d78ce08 not found: ID does not exist"
Oct 30 21:39:04.461047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.461002    6695 scope.go:122] "RemoveContainer" containerID="8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"
Oct 30 21:39:04.461299 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:39:04.461282    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440\": container with ID starting with 8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440 not found: ID does not exist" containerID="8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"
Oct 30 21:39:04.461346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.461303    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440"} err="failed to get container status \"8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440\": rpc error: code = NotFound desc = could not find container \"8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440\": container with ID starting with 8bec219a89c5e402c6d3ebec1d487a15807c7d68b0bcff6bc89e4084067c5440 not found: ID does not exist"
Oct 30 21:39:04.487392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.487351    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"]
Oct 30 21:39:04.497452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.497413    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0e991-openshift-default-795d5cb6f7-47xg4"]
Oct 30 21:39:04.919521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.919488    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:39:04.994577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.994540    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:39:04.995187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:04.994663    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 30 21:39:05.016068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016024    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016089    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016171    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016204    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016261    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016295    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016342    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016378    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.016992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.016491    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx\") pod \"d446414a-be00-4d2d-97cc-006645cb67d8\" (UID: \"d446414a-be00-4d2d-97cc-006645cb67d8\") "
Oct 30 21:39:05.017109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.017077    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:05.017402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.017376    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:05.017489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.017467    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:05.017931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.017887    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:05.018116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.018084    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:05.020112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.019519    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:39:05.020676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.020634    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:05.020813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.020790    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "kube-api-access-5jnsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:05.021217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.021190    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token" pod "d446414a-be00-4d2d-97cc-006645cb67d8" (UID: "d446414a-be00-4d2d-97cc-006645cb67d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:05.053788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.053747    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"]
Oct 30 21:39:05.054553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054530    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy"
Oct 30 21:39:05.054595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054560    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy"
Oct 30 21:39:05.054595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054579    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b5068dcf-7769-4224-81aa-a1b2189001a8" containerName="revoke-keys"
Oct 30 21:39:05.054595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054587    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5068dcf-7769-4224-81aa-a1b2189001a8" containerName="revoke-keys"
Oct 30 21:39:05.054743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054651    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy"
Oct 30 21:39:05.054743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054663    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy"
Oct 30 21:39:05.054743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054680    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy"
Oct 30 21:39:05.054743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054688    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy"
Oct 30 21:39:05.054866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054838    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy"
Oct 30 21:39:05.054901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054865    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy"
Oct 30 21:39:05.054901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054877    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5068dcf-7769-4224-81aa-a1b2189001a8" containerName="revoke-keys"
Oct 30 21:39:05.054901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.054890    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy"
Oct 30 21:39:05.059198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.059160    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.092136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.092057    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:05.092349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.092220    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"]
Oct 30 21:39:05.094411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.094379    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:05.094684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.094662    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:05.118508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118468    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jnsx\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-kube-api-access-5jnsx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118505    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446414a-be00-4d2d-97cc-006645cb67d8-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118518    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118529    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118541    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118552    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446414a-be00-4d2d-97cc-006645cb67d8-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118564    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118576    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446414a-be00-4d2d-97cc-006645cb67d8-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.118822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.118660    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446414a-be00-4d2d-97cc-006645cb67d8-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:05.191263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.191234    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:05.214871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.214784    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" path="/var/lib/kubelet/pods/7c1730f8-e0e3-4345-9952-cde0a41a63b5/volumes"
Oct 30 21:39:05.219206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.219166    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8n9d\" (UniqueName: \"kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.219421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.219306    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.219421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.219346    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.320586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.320533    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8n9d\" (UniqueName: \"kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.320856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.320701    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.320856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.320793    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.321514 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.321493    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.323425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.323405    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.339447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.339405    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8n9d\" (UniqueName: \"kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.378297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.378253    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:05.455650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.455066    6695 generic.go:356] "Generic (PLEG): container finished" podID="d446414a-be00-4d2d-97cc-006645cb67d8" containerID="5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0" exitCode=0
Oct 30 21:39:05.456004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.455982    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"
Oct 30 21:39:05.456535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.456507    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" event={"ID":"d446414a-be00-4d2d-97cc-006645cb67d8","Type":"ContainerDied","Data":"5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0"}
Oct 30 21:39:05.456699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.456562    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2" event={"ID":"d446414a-be00-4d2d-97cc-006645cb67d8","Type":"ContainerDied","Data":"c4212339c1a77544f025a1eee43a10ba3340f052392cfe5bacb874becbae8580"}
Oct 30 21:39:05.456699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.456586    6695 scope.go:122] "RemoveContainer" containerID="5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0"
Oct 30 21:39:05.469165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.469142    6695 scope.go:122] "RemoveContainer" containerID="b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"
Oct 30 21:39:05.482496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.482387    6695 scope.go:122] "RemoveContainer" containerID="5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0"
Oct 30 21:39:05.482853 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:39:05.482829    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0\": container with ID starting with 5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0 not found: ID does not exist" containerID="5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0"
Oct 30 21:39:05.482944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.482869    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0"} err="failed to get container status \"5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0\": rpc error: code = NotFound desc = could not find container \"5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0\": container with ID starting with 5c4d405d6ca2a520d05f2a626ab1bf1974db2bf1be525a6a8738fdd7c23628a0 not found: ID does not exist"
Oct 30 21:39:05.482944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.482894    6695 scope.go:122] "RemoveContainer" containerID="b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"
Oct 30 21:39:05.483249 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:39:05.483227    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5\": container with ID starting with b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5 not found: ID does not exist" containerID="b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"
Oct 30 21:39:05.483325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.483256    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5"} err="failed to get container status \"b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5\": rpc error: code = NotFound desc = could not find container \"b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5\": container with ID starting with b6b12cdf060166333765fad9fd67696ec97f82c9b19660c431257eaf2e604bb5 not found: ID does not exist"
Oct 30 21:39:05.508986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.508946    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"]
Oct 30 21:39:05.516452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.516418    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-912e61-openshift-default-6996ffcb45-d7rc2"]
Oct 30 21:39:05.687075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:05.687042    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"]
Oct 30 21:39:05.689546 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:05.689511    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af5ab47_6cbc_4de7_a86b_935625634478.slice/crio-d333001c4d6edd2ecd89224251553a6548c417ed4af80bf0699adf33d226737c WatchSource:0}: Error finding container d333001c4d6edd2ecd89224251553a6548c417ed4af80bf0699adf33d226737c: Status 404 returned error can't find the container with id d333001c4d6edd2ecd89224251553a6548c417ed4af80bf0699adf33d226737c
Oct 30 21:39:06.445348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:06.445248    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:39:06.445884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:06.445862    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:39:06.447173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:06.447149    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:39:06.470238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:06.470186    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t" event={"ID":"7af5ab47-6cbc-4de7-a86b-935625634478","Type":"ContainerStarted","Data":"eb795d8e9fd8fce557cabbaa23276008398e108a5ed36296741c1a256b5927c0"}
Oct 30 21:39:06.470456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:06.470265    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t" event={"ID":"7af5ab47-6cbc-4de7-a86b-935625634478","Type":"ContainerStarted","Data":"d333001c4d6edd2ecd89224251553a6548c417ed4af80bf0699adf33d226737c"}
Oct 30 21:39:06.570518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:06.570455    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t" podStartSLOduration=1.570437306 podStartE2EDuration="1.570437306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:06.568968978 +0000 UTC m=+1729.491427276" watchObservedRunningTime="2026-10-30 21:39:06.570437306 +0000 UTC m=+1729.492895610"
Oct 30 21:39:07.145927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.145879    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"]
Oct 30 21:39:07.146449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.146433    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy"
Oct 30 21:39:07.146493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.146451    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" containerName="istio-proxy"
Oct 30 21:39:07.146556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.146545    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1730f8-e0e3-4345-9952-cde0a41a63b5" containerName="istio-proxy"
Oct 30 21:39:07.149844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.149817    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.169426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.169384    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"]
Oct 30 21:39:07.215433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.215394    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d446414a-be00-4d2d-97cc-006645cb67d8" path="/var/lib/kubelet/pods/d446414a-be00-4d2d-97cc-006645cb67d8/volumes"
Oct 30 21:39:07.243501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.243460    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.243711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.243512    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.243758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.243737    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh86\" (UniqueName: \"kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.345044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.345002    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh86\" (UniqueName: \"kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.345209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.345100    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.345209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.345147    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.345913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.345878    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.347575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.347554    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.366818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.366780    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh86\" (UniqueName: \"kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86\") pod \"maas-api-revoke-sub-2dbd3d78-886d4\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.465808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.465719    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:07.484057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.484024    6695 generic.go:356] "Generic (PLEG): container finished" podID="7af5ab47-6cbc-4de7-a86b-935625634478" containerID="eb795d8e9fd8fce557cabbaa23276008398e108a5ed36296741c1a256b5927c0" exitCode=0
Oct 30 21:39:07.484202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.484108    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t" event={"ID":"7af5ab47-6cbc-4de7-a86b-935625634478","Type":"ContainerDied","Data":"eb795d8e9fd8fce557cabbaa23276008398e108a5ed36296741c1a256b5927c0"}
Oct 30 21:39:07.626183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.626146    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"]
Oct 30 21:39:07.631994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.631965    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.646245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.646216    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c19b75-dockercfg-n5hdc\""
Oct 30 21:39:07.646245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.646231    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c19b75\""
Oct 30 21:39:07.649702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.649578    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"]
Oct 30 21:39:07.739799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.739598    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"]
Oct 30 21:39:07.751930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.751872    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:07.765270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.765225    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.765474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.765296    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ks89\" (UniqueName: \"kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89\") pod \"payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.776156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.776116    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"]
Oct 30 21:39:07.812260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.812226    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"]
Oct 30 21:39:07.812571 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:07.812545    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef19ce3b_8bc8_4409_a1b6_33722f8d6da1.slice/crio-1e52454cb17b1d4ecc7bf5ed5a165d83d947e481056a0673071e7c14185e7818 WatchSource:0}: Error finding container 1e52454cb17b1d4ecc7bf5ed5a165d83d947e481056a0673071e7c14185e7818: Status 404 returned error can't find the container with id 1e52454cb17b1d4ecc7bf5ed5a165d83d947e481056a0673071e7c14185e7818
Oct 30 21:39:07.867102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.867051    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:07.867304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.867123    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhdl\" (UniqueName: \"kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl\") pod \"payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:07.867304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.867222    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.867304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.867263    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ks89\" (UniqueName: \"kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89\") pod \"payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.867998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.867971    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.885891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.885856    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ks89\" (UniqueName: \"kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89\") pod \"payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.949265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.949219    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:07.968037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.968002    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:07.968227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.968064    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhdl\" (UniqueName: \"kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl\") pod \"payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:07.968930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.968902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:07.986364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:07.986326    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhdl\" (UniqueName: \"kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl\") pod \"payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:08.094842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.094802    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:08.257990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.257955    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"]
Oct 30 21:39:08.262336 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:08.262296    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9444f0d1_2133_48b7_8121_c44255fd2155.slice/crio-a02c60cf54664708d92aad0c632fbdbcd3396eef1f71554d2391a76eb79906fb WatchSource:0}: Error finding container a02c60cf54664708d92aad0c632fbdbcd3396eef1f71554d2391a76eb79906fb: Status 404 returned error can't find the container with id a02c60cf54664708d92aad0c632fbdbcd3396eef1f71554d2391a76eb79906fb
Oct 30 21:39:08.416224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.416182    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"]
Oct 30 21:39:08.494021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.493982    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" event={"ID":"9444f0d1-2133-48b7-8121-c44255fd2155","Type":"ContainerStarted","Data":"712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"}
Oct 30 21:39:08.494669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.494080    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" event={"ID":"9444f0d1-2133-48b7-8121-c44255fd2155","Type":"ContainerStarted","Data":"a02c60cf54664708d92aad0c632fbdbcd3396eef1f71554d2391a76eb79906fb"}
Oct 30 21:39:08.494669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.494254    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:08.495648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.495619    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" event={"ID":"a7681615-c61a-41e6-90f3-c2132a3ed797","Type":"ContainerStarted","Data":"84893528202fe04b347ea4abeb38e67784aa3271f9268ba1cb90b0fd4a55a8dd"}
Oct 30 21:39:08.497583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.497556    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4" event={"ID":"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1","Type":"ContainerStarted","Data":"bbde2636c1001b928852ed653165f982f4107592f17a10e7077ed3cbca09e63d"}
Oct 30 21:39:08.497716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.497621    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4" event={"ID":"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1","Type":"ContainerStarted","Data":"1e52454cb17b1d4ecc7bf5ed5a165d83d947e481056a0673071e7c14185e7818"}
Oct 30 21:39:08.536742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.536508    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" podStartSLOduration=1.536485646 podStartE2EDuration="1.536485646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:08.531100882 +0000 UTC m=+1731.453559181" watchObservedRunningTime="2026-10-30 21:39:08.536485646 +0000 UTC m=+1731.458943946"
Oct 30 21:39:08.586636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.579883    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4" podStartSLOduration=1.579863337 podStartE2EDuration="1.579863337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:08.568330742 +0000 UTC m=+1731.490789044" watchObservedRunningTime="2026-10-30 21:39:08.579863337 +0000 UTC m=+1731.502321636"
Oct 30 21:39:08.698126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.698091    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:08.785852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.785800    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca\") pod \"7af5ab47-6cbc-4de7-a86b-935625634478\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") "
Oct 30 21:39:08.786048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.785911    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca\") pod \"7af5ab47-6cbc-4de7-a86b-935625634478\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") "
Oct 30 21:39:08.786164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.786144    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d\" (UniqueName: \"kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d\") pod \"7af5ab47-6cbc-4de7-a86b-935625634478\" (UID: \"7af5ab47-6cbc-4de7-a86b-935625634478\") "
Oct 30 21:39:08.786209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.786182    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca" pod "7af5ab47-6cbc-4de7-a86b-935625634478" (UID: "7af5ab47-6cbc-4de7-a86b-935625634478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:08.786540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.786514    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:08.788065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.787998    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca" pod "7af5ab47-6cbc-4de7-a86b-935625634478" (UID: "7af5ab47-6cbc-4de7-a86b-935625634478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:08.788165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.788116    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d" pod "7af5ab47-6cbc-4de7-a86b-935625634478" (UID: "7af5ab47-6cbc-4de7-a86b-935625634478"). InnerVolumeSpecName "kube-api-access-w8n9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:08.887357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.887316    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8n9d\" (UniqueName: \"kubernetes.io/projected/7af5ab47-6cbc-4de7-a86b-935625634478-kube-api-access-w8n9d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:08.887357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:08.887354    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af5ab47-6cbc-4de7-a86b-935625634478-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:09.506863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.506765    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" event={"ID":"a7681615-c61a-41e6-90f3-c2132a3ed797","Type":"ContainerStarted","Data":"c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2"}
Oct 30 21:39:09.506863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.506854    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:09.508553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.508528    6695 generic.go:356] "Generic (PLEG): container finished" podID="ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" containerID="bbde2636c1001b928852ed653165f982f4107592f17a10e7077ed3cbca09e63d" exitCode=0
Oct 30 21:39:09.508677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.508618    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4" event={"ID":"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1","Type":"ContainerDied","Data":"bbde2636c1001b928852ed653165f982f4107592f17a10e7077ed3cbca09e63d"}
Oct 30 21:39:09.510295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.510275    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"
Oct 30 21:39:09.510502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.510313    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t" event={"ID":"7af5ab47-6cbc-4de7-a86b-935625634478","Type":"ContainerDied","Data":"d333001c4d6edd2ecd89224251553a6548c417ed4af80bf0699adf33d226737c"}
Oct 30 21:39:09.510502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.510358    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d333001c4d6edd2ecd89224251553a6548c417ed4af80bf0699adf33d226737c"
Oct 30 21:39:09.550316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:09.550258    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" podStartSLOduration=2.550241833 podStartE2EDuration="2.550241833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:09.547770495 +0000 UTC m=+1732.470228795" watchObservedRunningTime="2026-10-30 21:39:09.550241833 +0000 UTC m=+1732.472700131"
Oct 30 21:39:10.664867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.664828    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:10.717632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.717589    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca\") pod \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") "
Oct 30 21:39:10.717788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.717664    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca\") pod \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") "
Oct 30 21:39:10.717853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.717818    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86\" (UniqueName: \"kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86\") pod \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\" (UID: \"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1\") "
Oct 30 21:39:10.718107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.718070    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca" pod "ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" (UID: "ef19ce3b-8bc8-4409-a1b6-33722f8d6da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:10.718252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.718237    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:10.719858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.719824    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca" pod "ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" (UID: "ef19ce3b-8bc8-4409-a1b6-33722f8d6da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:10.719996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.719971    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86" pod "ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" (UID: "ef19ce3b-8bc8-4409-a1b6-33722f8d6da1"). InnerVolumeSpecName "kube-api-access-hgh86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:10.819580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.819545    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgh86\" (UniqueName: \"kubernetes.io/projected/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-kube-api-access-hgh86\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:10.819580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:10.819575    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:11.526373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.526343    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"
Oct 30 21:39:11.526553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.526374    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4" event={"ID":"ef19ce3b-8bc8-4409-a1b6-33722f8d6da1","Type":"ContainerDied","Data":"1e52454cb17b1d4ecc7bf5ed5a165d83d947e481056a0673071e7c14185e7818"}
Oct 30 21:39:11.526553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.526425    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e52454cb17b1d4ecc7bf5ed5a165d83d947e481056a0673071e7c14185e7818"
Oct 30 21:39:11.990371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990293    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"]
Oct 30 21:39:11.990846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990832    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7af5ab47-6cbc-4de7-a86b-935625634478" containerName="revoke-keys"
Oct 30 21:39:11.990888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990847    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af5ab47-6cbc-4de7-a86b-935625634478" containerName="revoke-keys"
Oct 30 21:39:11.990888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990856    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" containerName="revoke-keys"
Oct 30 21:39:11.990888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990861    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" containerName="revoke-keys"
Oct 30 21:39:11.990983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990945    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af5ab47-6cbc-4de7-a86b-935625634478" containerName="revoke-keys"
Oct 30 21:39:11.990983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.990954    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" containerName="revoke-keys"
Oct 30 21:39:11.994401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:11.994387    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.010684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.010663    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:12.010793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.010691    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:12.022164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.022138    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:12.022164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.022148    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:12.026649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.026617    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"]
Oct 30 21:39:12.132818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.132779    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.133007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.132923    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.133007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.132993    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csltj\" (UniqueName: \"kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.234683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.234648    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.234874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.234750    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.234874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.234818    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csltj\" (UniqueName: \"kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.235307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.235272    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.237056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.237036    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.253794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.253731    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csltj\" (UniqueName: \"kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj\") pod \"maas-api-revoke-sub-90915438-6mp2c\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.307629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.307579    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:12.614662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.614626    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"]
Oct 30 21:39:12.618062 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:12.618020    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8928e6d3_a3b3_4e67_92f7_2f08c16806a3.slice/crio-519b3a35fd6009783f2ff1e0e92cc6afc1b46ebbc6570af2c7f3e0c218cf47dd WatchSource:0}: Error finding container 519b3a35fd6009783f2ff1e0e92cc6afc1b46ebbc6570af2c7f3e0c218cf47dd: Status 404 returned error can't find the container with id 519b3a35fd6009783f2ff1e0e92cc6afc1b46ebbc6570af2c7f3e0c218cf47dd
Oct 30 21:39:12.718808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.718423    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"]
Oct 30 21:39:12.723853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.723836    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.754965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.754931    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"]
Oct 30 21:39:12.842985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.842954    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.842993    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843085    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843114    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843200    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843283    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgb4\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843373    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843419    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.843546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.843533    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.944959    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945014    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945098    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945124    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945141    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945180    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945205    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945261    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgb4\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945392    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945436    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945474    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.945807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.945522    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.946097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.946073    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.947531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.947510    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.947690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.947672    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.964898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.964855    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:12.965184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:12.965162    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgb4\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:13.039649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.039591    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:13.358833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.358801    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"]
Oct 30 21:39:13.362546 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:13.362517    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e413a6_fdf1_45ca_ba7c_f4d051d9dfd6.slice/crio-ba24d2375f9b8aab6f118b9d9d0ea00967c31018223dbd6f4ee5c7206f652b27 WatchSource:0}: Error finding container ba24d2375f9b8aab6f118b9d9d0ea00967c31018223dbd6f4ee5c7206f652b27: Status 404 returned error can't find the container with id ba24d2375f9b8aab6f118b9d9d0ea00967c31018223dbd6f4ee5c7206f652b27
Oct 30 21:39:13.365054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.365023    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:13.365161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.365099    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:13.365161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.365128    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:13.545455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.545429    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" event={"ID":"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6","Type":"ContainerStarted","Data":"465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb"}
Oct 30 21:39:13.545659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.545633    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" event={"ID":"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6","Type":"ContainerStarted","Data":"ba24d2375f9b8aab6f118b9d9d0ea00967c31018223dbd6f4ee5c7206f652b27"}
Oct 30 21:39:13.547442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.547419    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c" event={"ID":"8928e6d3-a3b3-4e67-92f7-2f08c16806a3","Type":"ContainerStarted","Data":"6a1f77393c35840ef966a45b9afcb7f10d1dbc36728ed740d5b6fc09f712e50e"}
Oct 30 21:39:13.547554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.547463    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c" event={"ID":"8928e6d3-a3b3-4e67-92f7-2f08c16806a3","Type":"ContainerStarted","Data":"519b3a35fd6009783f2ff1e0e92cc6afc1b46ebbc6570af2c7f3e0c218cf47dd"}
Oct 30 21:39:13.590267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:13.590207    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podStartSLOduration=1.590189 podStartE2EDuration="1.590189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:13.586060599 +0000 UTC m=+1736.508518897" watchObservedRunningTime="2026-10-30 21:39:13.590189 +0000 UTC m=+1736.512647299"
Oct 30 21:39:14.040332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.040294    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:14.042348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.042319    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:14.042495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.042375    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:14.559018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.558981    6695 generic.go:356] "Generic (PLEG): container finished" podID="8928e6d3-a3b3-4e67-92f7-2f08c16806a3" containerID="6a1f77393c35840ef966a45b9afcb7f10d1dbc36728ed740d5b6fc09f712e50e" exitCode=0
Oct 30 21:39:14.559211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.559066    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c" event={"ID":"8928e6d3-a3b3-4e67-92f7-2f08c16806a3","Type":"ContainerDied","Data":"6a1f77393c35840ef966a45b9afcb7f10d1dbc36728ed740d5b6fc09f712e50e"}
Oct 30 21:39:14.906524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.906483    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:14.987983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.987947    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca\") pod \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") "
Oct 30 21:39:14.988168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.988027    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca\") pod \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") "
Oct 30 21:39:14.988259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.988240    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj\" (UniqueName: \"kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj\") pod \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\" (UID: \"8928e6d3-a3b3-4e67-92f7-2f08c16806a3\") "
Oct 30 21:39:14.988513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.988474    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca" pod "8928e6d3-a3b3-4e67-92f7-2f08c16806a3" (UID: "8928e6d3-a3b3-4e67-92f7-2f08c16806a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.988850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.988826    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:14.990524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.990485    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj" pod "8928e6d3-a3b3-4e67-92f7-2f08c16806a3" (UID: "8928e6d3-a3b3-4e67-92f7-2f08c16806a3"). InnerVolumeSpecName "kube-api-access-csltj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.993270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:14.993240    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca" pod "8928e6d3-a3b3-4e67-92f7-2f08c16806a3" (UID: "8928e6d3-a3b3-4e67-92f7-2f08c16806a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:15.054484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.054025    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:15.054484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.054089    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:15.090014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.089870    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csltj\" (UniqueName: \"kubernetes.io/projected/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-kube-api-access-csltj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:15.090014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.089919    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8928e6d3-a3b3-4e67-92f7-2f08c16806a3-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:15.379290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.379241    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"]
Oct 30 21:39:15.379845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.379819    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8928e6d3-a3b3-4e67-92f7-2f08c16806a3" containerName="revoke-keys"
Oct 30 21:39:15.379845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.379837    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8928e6d3-a3b3-4e67-92f7-2f08c16806a3" containerName="revoke-keys"
Oct 30 21:39:15.380068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.379942    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8928e6d3-a3b3-4e67-92f7-2f08c16806a3" containerName="revoke-keys"
Oct 30 21:39:15.383537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.383513    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.396445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.396415    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"]
Oct 30 21:39:15.401260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.401228    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.402372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.402344    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-49738b-openshift-default-dockercfg-h4mdb\""
Oct 30 21:39:15.426973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.426898    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"]
Oct 30 21:39:15.462477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.460042    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"]
Oct 30 21:39:15.495520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.495762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495541    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.495762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495577    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.495762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495651    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.495897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495757    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.495897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495798    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.495897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495857    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.495897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495887    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.496028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495917    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.496028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495949    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.496028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.495985    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p722x\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.496128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496086    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r29\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.496166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496130    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.496166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496155    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.496250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496235    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.496288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496257    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.496326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496292    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.496326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.496310    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.573410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.573377    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"
Oct 30 21:39:15.574186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.574162    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c" event={"ID":"8928e6d3-a3b3-4e67-92f7-2f08c16806a3","Type":"ContainerDied","Data":"519b3a35fd6009783f2ff1e0e92cc6afc1b46ebbc6570af2c7f3e0c218cf47dd"}
Oct 30 21:39:15.574381 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.574364    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="519b3a35fd6009783f2ff1e0e92cc6afc1b46ebbc6570af2c7f3e0c218cf47dd"
Oct 30 21:39:15.597687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.597664    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.597814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.597697    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.597814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.597719    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.597970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.597945    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.597991    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598031    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598052    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598055    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598106    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p722x\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598193    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r29\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598222    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598338    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598363    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598419    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598452    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598485    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.598595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598549    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598587    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598792    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.598960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.598937    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.599131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.599108    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.599405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.599379    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.599485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.599404    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.599539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.599506    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.599636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.599387    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.599721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.599701    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.600092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.600072    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.601245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.601226    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.601432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.601411    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.601706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.601685    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.601900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.601885    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.649409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.647136    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.649409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.649337    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.650841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.650813    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p722x\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:15.651089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.651068    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r29\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.703320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.703242    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:15.747628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:15.746836    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:16.044632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.040009    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:16.044632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.040065    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:16.049966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.049932    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"]
Oct 30 21:39:16.087284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.087255    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"]
Oct 30 21:39:16.088260 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:16.088225    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3a6a78_9914_46c0_b85b_6efb55452ade.slice/crio-51daf71d6ae67c916f138d616f673a86c71666c51c5265dc47580f5d39d7a328 WatchSource:0}: Error finding container 51daf71d6ae67c916f138d616f673a86c71666c51c5265dc47580f5d39d7a328: Status 404 returned error can't find the container with id 51daf71d6ae67c916f138d616f673a86c71666c51c5265dc47580f5d39d7a328
Oct 30 21:39:16.091912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.091852    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.092019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.091944    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.092019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.091993    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.097011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.096982    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"]
Oct 30 21:39:16.104823 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:16.104796    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda287432e_f634_4fbd_bb68_7e04f867f6c2.slice/crio-3873af55b32c71f8f17b1b7e104a6b1c1ae67fba5a2444df389b5692bdb41f7c WatchSource:0}: Error finding container 3873af55b32c71f8f17b1b7e104a6b1c1ae67fba5a2444df389b5692bdb41f7c: Status 404 returned error can't find the container with id 3873af55b32c71f8f17b1b7e104a6b1c1ae67fba5a2444df389b5692bdb41f7c
Oct 30 21:39:16.106849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.106819    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.106960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.106883    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.106960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.106914    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.592092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.592055    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" event={"ID":"ea3a6a78-9914-46c0-b85b-6efb55452ade","Type":"ContainerStarted","Data":"ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8"}
Oct 30 21:39:16.592092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.592112    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" event={"ID":"ea3a6a78-9914-46c0-b85b-6efb55452ade","Type":"ContainerStarted","Data":"51daf71d6ae67c916f138d616f673a86c71666c51c5265dc47580f5d39d7a328"}
Oct 30 21:39:16.594623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.594513    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" event={"ID":"a287432e-f634-4fbd-bb68-7e04f867f6c2","Type":"ContainerStarted","Data":"ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc"}
Oct 30 21:39:16.594623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.594572    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" event={"ID":"a287432e-f634-4fbd-bb68-7e04f867f6c2","Type":"ContainerStarted","Data":"3873af55b32c71f8f17b1b7e104a6b1c1ae67fba5a2444df389b5692bdb41f7c"}
Oct 30 21:39:16.651810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.651748    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podStartSLOduration=1.6517298409999999 podStartE2EDuration="1.651729841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:16.650535565 +0000 UTC m=+1739.572993862" watchObservedRunningTime="2026-10-30 21:39:16.651729841 +0000 UTC m=+1739.574188140"
Oct 30 21:39:16.694888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.694831    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podStartSLOduration=1.694817604 podStartE2EDuration="1.694817604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:16.693758903 +0000 UTC m=+1739.616217200" watchObservedRunningTime="2026-10-30 21:39:16.694817604 +0000 UTC m=+1739.617275902"
Oct 30 21:39:16.703864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.703825    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:16.748138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:16.748092    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:17.042048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.042010    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:17.042230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.042070    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:17.706205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.706139    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:17.706823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.706258    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:17.754630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.753713    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:17.754630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.753781    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:17.870625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.857933    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"]
Oct 30 21:39:17.990933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.990844    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"]
Oct 30 21:39:17.991291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:17.991268    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.074627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.074091    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:18.074627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.074151    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076003    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076104    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076129    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076158    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076184    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zgp\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076228    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076272    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.085665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.076366    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177093    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177169    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177263    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177346    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177381    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177434    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177503    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.177526    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zgp\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.178316    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.191561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.191525    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.195630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.192071    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.218628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.196479    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.218919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.200554    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.231743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.223926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zgp\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.231743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.226993    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.296250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.278280    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.296250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.291154    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.403734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.403701    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:18.762876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.761032    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:18.762876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.761101    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:18.779628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.769818    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:18.789629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:18.780499    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:19.092655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.092319    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:19.092655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.092378    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:19.172628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.151208    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"]
Oct 30 21:39:19.193175 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:19.193138    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05f24da_31fd_4428_82f1_42fbe7f7615b.slice/crio-516a4c4c58aab084f79e76096dc0e306214ab02ed45f3f641e348f6cfdc92466 WatchSource:0}: Error finding container 516a4c4c58aab084f79e76096dc0e306214ab02ed45f3f641e348f6cfdc92466: Status 404 returned error can't find the container with id 516a4c4c58aab084f79e76096dc0e306214ab02ed45f3f641e348f6cfdc92466
Oct 30 21:39:19.196082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.196048    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:19.196213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.196117    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:19.196213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.196145    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:19.557958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.557874    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:39:19.699101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.699063    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" event={"ID":"b05f24da-31fd-4428-82f1-42fbe7f7615b","Type":"ContainerStarted","Data":"a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066"}
Oct 30 21:39:19.699684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.699659    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" event={"ID":"b05f24da-31fd-4428-82f1-42fbe7f7615b","Type":"ContainerStarted","Data":"516a4c4c58aab084f79e76096dc0e306214ab02ed45f3f641e348f6cfdc92466"}
Oct 30 21:39:19.704857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.704440    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:19.704857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.704490    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:19.751625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.751041    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:19.751625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.751106    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:19.791771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:19.791589    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podStartSLOduration=2.791568132 podStartE2EDuration="2.791568132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:19.769556517 +0000 UTC m=+1742.692014816" watchObservedRunningTime="2026-10-30 21:39:19.791568132 +0000 UTC m=+1742.714026431"
Oct 30 21:39:20.040638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.040482    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:20.040638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.040569    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:20.404237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.404200    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:20.405668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.405642    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:20.405809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.405691    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:20.535625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.533831    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:39:20.795166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.750700    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:20.795939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.795904    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:20.796449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.796426    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:20.796656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.796633    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:20.883100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:20.883042    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"]
Oct 30 21:39:21.059718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.054631    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k" podUID="bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" containerName="payload-pre-processing" containerID="cri-o://436b4f95d036338c3ba6ff16ed16ae89a7e8fb152cd3e0f7ab38f8fcc36209dc" gracePeriod=25
Oct 30 21:39:21.065752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.065712    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:39:21.066020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.065991    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Oct 30 21:39:21.338090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.337998    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.387715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.367980    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:21.387715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.368116    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:21.387715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.368232    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:21.387715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.368305    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:21.445629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.409017    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"]
Oct 30 21:39:21.491868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.445527    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.491868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.482404    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhcv8\" (UniqueName: \"kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.491868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.482469    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.491868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.446185    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:21.491868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.482832    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:21.685628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.590660    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.704675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.697825    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.704675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.697873    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhcv8\" (UniqueName: \"kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.731280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.730688    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 30 21:39:21.797625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.796813    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.822631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.799081    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:21.822631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.799128    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:21.822631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.805986    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:21.822631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.802137    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:21.822631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.806737    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:21.940682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.940521    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:21.950655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.941761    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:39:21.983340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:21.983268    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhcv8\" (UniqueName: \"kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8\") pod \"maas-api-revoke-sub-09fe0be9-wncch\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:22.009881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.009848    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:22.030073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.030030    6695 generic.go:356] "Generic (PLEG): container finished" podID="bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" containerID="436b4f95d036338c3ba6ff16ed16ae89a7e8fb152cd3e0f7ab38f8fcc36209dc" exitCode=0
Oct 30 21:39:22.037633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.031503    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k" event={"ID":"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c","Type":"ContainerDied","Data":"436b4f95d036338c3ba6ff16ed16ae89a7e8fb152cd3e0f7ab38f8fcc36209dc"}
Oct 30 21:39:22.102794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.102739    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:22.436634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.434744    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:22.436634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.434809    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:22.822627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.805464    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:39:23.001680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.998328    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:39:23.001680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.998524    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:23.001680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.998535    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:39:23.001680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.998785    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:39:23.001680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:22.998848    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:39:23.007557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.007444    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.007557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.007489    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:23.102908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.054565    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.102908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.054643    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:23.102908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.057094    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:23.102908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.057139    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:23.183713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.183658    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:23.185143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.184260    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:39:23.340626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.339682    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:39:23.409627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.409385    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.409627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.409436    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:23.477074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.439298    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648\" (UniqueName: \"kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648\") pod \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") "
Oct 30 21:39:23.477074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.439365    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume\") pod \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\" (UID: \"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c\") "
Oct 30 21:39:23.490010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.489965    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume" pod "bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" (UID: "bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:23.495568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.495499    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:23.600452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.600416    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:39:23.600881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.600858    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:39:23.601032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.601014    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k" event={"ID":"bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c","Type":"ContainerDied","Data":"48ee3a86122284f814cf9983ceed7ffcac0ad9c7cdce055f1e0b9dd88f18d48a"}
Oct 30 21:39:23.601149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.601137    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"]
Oct 30 21:39:23.601278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.601269    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:39:23.602370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.602354    6695 scope.go:122] "RemoveContainer" containerID="436b4f95d036338c3ba6ff16ed16ae89a7e8fb152cd3e0f7ab38f8fcc36209dc"
Oct 30 21:39:23.703716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.703684    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.704107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.704073    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:23.731934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.702292    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648" pod "bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" (UID: "bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c"). InnerVolumeSpecName "kube-api-access-bw648". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:23.755742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.755682    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.756095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.756045    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:23.806996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:23.806970    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw648\" (UniqueName: \"kubernetes.io/projected/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c-kube-api-access-bw648\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:24.109797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.109758    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:24.110377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.110347    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:24.404410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.404371    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:24.404651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.404425    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:24.478590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.478551    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:24.479029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.479003    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:24.480119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.480096    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:39:24.569577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.566765    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch" event={"ID":"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a","Type":"ContainerStarted","Data":"cea834ce0c1617359a5163290c38a400315b6b244f56ee7c2b7a29d92ef4ba6c"}
Oct 30 21:39:24.569577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.566833    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch" event={"ID":"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a","Type":"ContainerStarted","Data":"f1205b3ffb4b52174ba04bf875afcba20bedffa9135a2fa29666f5105821bec0"}
Oct 30 21:39:24.572756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.572724    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"
Oct 30 21:39:24.637641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.637439    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch" podStartSLOduration=4.637415494 podStartE2EDuration="4.637415494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:24.608816808 +0000 UTC m=+1747.531275110" watchObservedRunningTime="2026-10-30 21:39:24.637415494 +0000 UTC m=+1747.559873793"
Oct 30 21:39:24.666311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.666209    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"]
Oct 30 21:39:24.673986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.672479    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e4693-f56df569f-h6b5k"]
Oct 30 21:39:24.706281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.706233    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Oct 30 21:39:24.706470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.706313    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Oct 30 21:39:24.748857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.748815    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:39:24.749064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:24.748899    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Oct 30 21:39:25.262632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.261916    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" path="/var/lib/kubelet/pods/bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c/volumes"
Oct 30 21:39:25.404872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.404822    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:25.405110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.404888    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:25.584288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.584248    6695 generic.go:356] "Generic (PLEG): container finished" podID="1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" containerID="cea834ce0c1617359a5163290c38a400315b6b244f56ee7c2b7a29d92ef4ba6c" exitCode=0
Oct 30 21:39:25.585651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.585590    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch" event={"ID":"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a","Type":"ContainerDied","Data":"cea834ce0c1617359a5163290c38a400315b6b244f56ee7c2b7a29d92ef4ba6c"}
Oct 30 21:39:25.708987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.708954    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:25.709281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.709259    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:25.710056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.710032    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:39:25.753854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.753817    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:25.754154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.754130    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:25.755121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:25.755100    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:39:26.404907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.404856    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:26.405313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.404923    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:26.469223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.469186    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"]
Oct 30 21:39:26.480130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.480082    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e4693-5h64t"]
Oct 30 21:39:26.837771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.837729    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:26.965567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.965527    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca\") pod \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") "
Oct 30 21:39:26.965831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.965595    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca\") pod \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") "
Oct 30 21:39:26.965831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.965678    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8\" (UniqueName: \"kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8\") pod \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\" (UID: \"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a\") "
Oct 30 21:39:26.966079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.966051    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca" pod "1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" (UID: "1ca2c8d5-5f36-4c08-9734-ef02b4a2862a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:26.966256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.966240    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:26.967940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.967905    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca" pod "1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" (UID: "1ca2c8d5-5f36-4c08-9734-ef02b4a2862a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:26.968088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:26.968067    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8" pod "1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" (UID: "1ca2c8d5-5f36-4c08-9734-ef02b4a2862a"). InnerVolumeSpecName "kube-api-access-jhcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:27.067320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.067275    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:27.067320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.067318    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhcv8\" (UniqueName: \"kubernetes.io/projected/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a-kube-api-access-jhcv8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:27.215453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.215401    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af5ab47-6cbc-4de7-a86b-935625634478" path="/var/lib/kubelet/pods/7af5ab47-6cbc-4de7-a86b-935625634478/volumes"
Oct 30 21:39:27.404411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.404320    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Oct 30 21:39:27.404411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.404388    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Oct 30 21:39:27.611030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.610990    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"
Oct 30 21:39:27.611030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.611005    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch" event={"ID":"1ca2c8d5-5f36-4c08-9734-ef02b4a2862a","Type":"ContainerDied","Data":"f1205b3ffb4b52174ba04bf875afcba20bedffa9135a2fa29666f5105821bec0"}
Oct 30 21:39:27.611666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:27.611098    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1205b3ffb4b52174ba04bf875afcba20bedffa9135a2fa29666f5105821bec0"
Oct 30 21:39:28.409801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:28.409768    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:28.410131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:28.410110    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:28.411022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:28.410995    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:39:29.872572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.872528    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"]
Oct 30 21:39:29.873393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.873366    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" containerName="revoke-keys"
Oct 30 21:39:29.873570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.873417    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" containerName="revoke-keys"
Oct 30 21:39:29.873570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.873449    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" containerName="payload-pre-processing"
Oct 30 21:39:29.873570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.873458    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" containerName="payload-pre-processing"
Oct 30 21:39:29.873796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.873637    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" containerName="revoke-keys"
Oct 30 21:39:29.873796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.873658    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc0dcfc-dec0-4aa8-8e3b-29eb79d2fc5c" containerName="payload-pre-processing"
Oct 30 21:39:29.877683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.877651    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:29.891925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.891887    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:29.903178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.903143    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:29.903405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.903347    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:29.903473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.903399    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:29.906785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:29.906746    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"]
Oct 30 21:39:30.008557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.008510    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.008805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.008570    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcfj\" (UniqueName: \"kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.008805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.008728    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.115500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.115432    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.116850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.116815    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcfj\" (UniqueName: \"kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.117197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.117174    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.118062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.116656    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.160252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.160130    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.161110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.161081    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcfj\" (UniqueName: \"kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj\") pod \"maas-api-revoke-sub-a5f3aaf7-2dsjb\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.193096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.193052    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:30.508111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.508028    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"]
Oct 30 21:39:30.512502 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:30.512469    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097cfe1d_c121_4908_bf27_be1986f66b69.slice/crio-93c9f49c8e6c0baa230e849f8a2305e160c24746c4fd1b041310af14c5856481 WatchSource:0}: Error finding container 93c9f49c8e6c0baa230e849f8a2305e160c24746c4fd1b041310af14c5856481: Status 404 returned error can't find the container with id 93c9f49c8e6c0baa230e849f8a2305e160c24746c4fd1b041310af14c5856481
Oct 30 21:39:30.638148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:30.638106    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb" event={"ID":"097cfe1d-c121-4908-bf27-be1986f66b69","Type":"ContainerStarted","Data":"93c9f49c8e6c0baa230e849f8a2305e160c24746c4fd1b041310af14c5856481"}
Oct 30 21:39:31.076100 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:39:31.076014    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:39:31.647989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:31.647952    6695 generic.go:356] "Generic (PLEG): container finished" podID="097cfe1d-c121-4908-bf27-be1986f66b69" containerID="c3d4bb68e8eef4d65026325cda6002276a25e814189b64a777265d6db6bbe39a" exitCode=0
Oct 30 21:39:31.648161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:31.648054    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb" event={"ID":"097cfe1d-c121-4908-bf27-be1986f66b69","Type":"ContainerDied","Data":"c3d4bb68e8eef4d65026325cda6002276a25e814189b64a777265d6db6bbe39a"}
Oct 30 21:39:32.828106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.828075    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:32.953650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.953525    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca\") pod \"097cfe1d-c121-4908-bf27-be1986f66b69\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") "
Oct 30 21:39:32.953650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.953593    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj\" (UniqueName: \"kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj\") pod \"097cfe1d-c121-4908-bf27-be1986f66b69\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") "
Oct 30 21:39:32.953928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.953809    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca\") pod \"097cfe1d-c121-4908-bf27-be1986f66b69\" (UID: \"097cfe1d-c121-4908-bf27-be1986f66b69\") "
Oct 30 21:39:32.954068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.954037    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca" pod "097cfe1d-c121-4908-bf27-be1986f66b69" (UID: "097cfe1d-c121-4908-bf27-be1986f66b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:32.954312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.954289    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:32.956017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.955983    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj" pod "097cfe1d-c121-4908-bf27-be1986f66b69" (UID: "097cfe1d-c121-4908-bf27-be1986f66b69"). InnerVolumeSpecName "kube-api-access-qmcfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:32.956170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:32.956026    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca" pod "097cfe1d-c121-4908-bf27-be1986f66b69" (UID: "097cfe1d-c121-4908-bf27-be1986f66b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:33.055528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:33.055483    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmcfj\" (UniqueName: \"kubernetes.io/projected/097cfe1d-c121-4908-bf27-be1986f66b69-kube-api-access-qmcfj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:33.055528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:33.055521    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097cfe1d-c121-4908-bf27-be1986f66b69-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:33.666081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:33.666049    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"
Oct 30 21:39:33.666081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:33.666063    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb" event={"ID":"097cfe1d-c121-4908-bf27-be1986f66b69","Type":"ContainerDied","Data":"93c9f49c8e6c0baa230e849f8a2305e160c24746c4fd1b041310af14c5856481"}
Oct 30 21:39:33.666328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:33.666114    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c9f49c8e6c0baa230e849f8a2305e160c24746c4fd1b041310af14c5856481"
Oct 30 21:39:35.005503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.005464    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"]
Oct 30 21:39:35.006068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.006049    6695 cpu_manager.go:414] "containerMap: removing container" podUID="097cfe1d-c121-4908-bf27-be1986f66b69" containerName="revoke-keys"
Oct 30 21:39:35.006119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.006068    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097cfe1d-c121-4908-bf27-be1986f66b69" containerName="revoke-keys"
Oct 30 21:39:35.006173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.006163    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="097cfe1d-c121-4908-bf27-be1986f66b69" containerName="revoke-keys"
Oct 30 21:39:35.009407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.009380    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.024771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.024735    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:35.035990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.035953    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:35.036270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.035954    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:35.036270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.036169    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:35.040561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.040521    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"]
Oct 30 21:39:35.089713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.089673    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.089940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.089779    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbw4\" (UniqueName: \"kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.089940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.089815    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.191440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.191396    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.192145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.192097    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbw4\" (UniqueName: \"kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.193115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.193087    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.197472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.196846    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.219707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.219667    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.222925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.222887    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbw4\" (UniqueName: \"kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4\") pod \"maas-api-revoke-sub-b5a613af-2k7rj\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.325724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.325678    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:35.632630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.632583    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"]
Oct 30 21:39:35.636083 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:35.636046    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77838753_5eee_48df_9d11_f63b230b6534.slice/crio-42274f15a41e79eb396016e0ebed0614e5d9430c7fffebb4fba2497b6455ab81 WatchSource:0}: Error finding container 42274f15a41e79eb396016e0ebed0614e5d9430c7fffebb4fba2497b6455ab81: Status 404 returned error can't find the container with id 42274f15a41e79eb396016e0ebed0614e5d9430c7fffebb4fba2497b6455ab81
Oct 30 21:39:35.682788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:35.682749    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj" event={"ID":"77838753-5eee-48df-9d11-f63b230b6534","Type":"ContainerStarted","Data":"42274f15a41e79eb396016e0ebed0614e5d9430c7fffebb4fba2497b6455ab81"}
Oct 30 21:39:36.695013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.694974    6695 generic.go:356] "Generic (PLEG): container finished" podID="77838753-5eee-48df-9d11-f63b230b6534" containerID="0821ad68a07a0d446fa9484ff95f34a127cc823a334613b4fcdc4c1cb7dc9483" exitCode=0
Oct 30 21:39:36.695467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.695057    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj" event={"ID":"77838753-5eee-48df-9d11-f63b230b6534","Type":"ContainerDied","Data":"0821ad68a07a0d446fa9484ff95f34a127cc823a334613b4fcdc4c1cb7dc9483"}
Oct 30 21:39:36.859115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.859071    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"]
Oct 30 21:39:36.863454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.863419    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.894662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.894539    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7a4de0-openshift-default-dockercfg-6qmrc\""
Oct 30 21:39:36.902917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.902874    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"]
Oct 30 21:39:36.975730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975680    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.975970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975751    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.975970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975794    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.975970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975821    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.975970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975858    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.975970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975898    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6474g\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.975970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.975956    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.976345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.976106    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:36.976345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:36.976160    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.078386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.078350    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.078715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.078686    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.078883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.078866    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.080654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.079001    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.080848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.080827    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.081031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.081012    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6474g\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.081220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.081194    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.081539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.081512    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.081746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.081719    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.082461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.079288    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.083643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.079519    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.083805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.083236    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.084204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.083551    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.084345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.083631    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.084487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.083945    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.100633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.097171    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.108639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.108385    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6474g\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.114634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.113432    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token\") pod \"e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") " pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.192027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.186884    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:37.692638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.683563    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"]
Oct 30 21:39:37.708716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.708614    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:37.709280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.708721    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:37.709280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.708764    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:37.729392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:37.729343    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" event={"ID":"2c7f4209-3fba-41a2-b098-b81984be14cf","Type":"ContainerStarted","Data":"9750eaaaf96dc9b63bdd58a03b7e482ac2f02d001a865d757ece3e62bc6669d0"}
Oct 30 21:39:38.481696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.481651    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:38.539751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.538389    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca\") pod \"77838753-5eee-48df-9d11-f63b230b6534\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") "
Oct 30 21:39:38.539751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.538513    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca\") pod \"77838753-5eee-48df-9d11-f63b230b6534\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") "
Oct 30 21:39:38.539751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.538588    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4\" (UniqueName: \"kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4\") pod \"77838753-5eee-48df-9d11-f63b230b6534\" (UID: \"77838753-5eee-48df-9d11-f63b230b6534\") "
Oct 30 21:39:38.546350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.544146    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca" pod "77838753-5eee-48df-9d11-f63b230b6534" (UID: "77838753-5eee-48df-9d11-f63b230b6534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:38.611992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.591796    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca" pod "77838753-5eee-48df-9d11-f63b230b6534" (UID: "77838753-5eee-48df-9d11-f63b230b6534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:38.611992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.592457    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4" pod "77838753-5eee-48df-9d11-f63b230b6534" (UID: "77838753-5eee-48df-9d11-f63b230b6534"). InnerVolumeSpecName "kube-api-access-gvbw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:38.703717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.685672    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77838753-5eee-48df-9d11-f63b230b6534-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:38.703717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.685721    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77838753-5eee-48df-9d11-f63b230b6534-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:38.703717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.685740    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbw4\" (UniqueName: \"kubernetes.io/projected/77838753-5eee-48df-9d11-f63b230b6534-kube-api-access-gvbw4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:38.804630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.801870    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj" event={"ID":"77838753-5eee-48df-9d11-f63b230b6534","Type":"ContainerDied","Data":"42274f15a41e79eb396016e0ebed0614e5d9430c7fffebb4fba2497b6455ab81"}
Oct 30 21:39:38.804630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.801944    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42274f15a41e79eb396016e0ebed0614e5d9430c7fffebb4fba2497b6455ab81"
Oct 30 21:39:38.804630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.802025    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"
Oct 30 21:39:38.869163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:38.869107    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" event={"ID":"2c7f4209-3fba-41a2-b098-b81984be14cf","Type":"ContainerStarted","Data":"2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"}
Oct 30 21:39:39.199615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.199513    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podStartSLOduration=3.199485657 podStartE2EDuration="3.199485657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:39.079714586 +0000 UTC m=+1762.002172881" watchObservedRunningTime="2026-10-30 21:39:39.199485657 +0000 UTC m=+1762.121943958"
Oct 30 21:39:39.366843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.364806    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:39.366843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.364958    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:39.522631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.502717    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:39.542811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.542696    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:39.565644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.561998    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:39.813633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.812671    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:39:39.836186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.828572    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:39:39.931630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.930669    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:39.949776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:39.939961    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:39:40.293634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.269743    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:40.293634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.269817    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.392631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.391136    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.516183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.513708    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.516183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.514374    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:39:40.571989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.571878    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.596842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.596760    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:39:40.598235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.572673    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.598235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.597474    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:39:40.827129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.826996    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.828191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.828167    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:39:40.830313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.830282    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.858153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:40.858117    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:39:41.423823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.423769    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:41.526633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.514373    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:39:41.591659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.579777    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:39:41.600786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.600725    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:39:41.602079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.602044    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:39:41.679696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.676768    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:41.825632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.794702    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:41.825994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.825948    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:41.847769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.839711    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:39:41.855664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.852250    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:41.855664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.852326    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:41.902760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:41.902709    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:42.365911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.365875    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:39:42.493505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.480060    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:39:42.609670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.596674    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:39:42.609670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.603806    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:39:42.609670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.604768    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:42.657692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.656824    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:42.810450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.810408    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:39:42.878798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.878760    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:39:42.880440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.880406    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:39:42.884432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:42.884392    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:39:43.995444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:43.995399    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"]
Oct 30 21:39:43.996131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:43.996108    6695 cpu_manager.go:414] "containerMap: removing container" podUID="77838753-5eee-48df-9d11-f63b230b6534" containerName="revoke-keys"
Oct 30 21:39:43.996224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:43.996132    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77838753-5eee-48df-9d11-f63b230b6534" containerName="revoke-keys"
Oct 30 21:39:43.996320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:43.996306    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="77838753-5eee-48df-9d11-f63b230b6534" containerName="revoke-keys"
Oct 30 21:39:44.001205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.001172    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.019260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.019211    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:44.030315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.029351    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:44.049001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.048963    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:44.049195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.049166    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:44.052541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.052504    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"]
Oct 30 21:39:44.088617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.088551    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.088817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.088709    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.088817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.088740    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h628f\" (UniqueName: \"kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.189940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.189899    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.190126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.189954    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h628f\" (UniqueName: \"kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.190216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.190174    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.190903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.190870    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.192888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.192843    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.237374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.237332    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h628f\" (UniqueName: \"kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f\") pod \"maas-api-revoke-sub-6416a132-m4xzx\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.336955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.336915    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:44.456486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.456443    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"]
Oct 30 21:39:44.462174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.462134    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.485635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.485583    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7a4de0\""
Oct 30 21:39:44.492948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.492906    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"]
Oct 30 21:39:44.494722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.494664    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"]
Oct 30 21:39:44.496188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.495913    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgp2\" (UniqueName: \"kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2\") pod \"payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.496188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.496059    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.500207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.500165    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.545941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.545866    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7a4de0-dockercfg-22b88\""
Oct 30 21:39:44.644237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.599659    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"]
Oct 30 21:39:44.674817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.673883    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltp2\" (UniqueName: \"kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2\") pod \"payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.674817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.674056    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgp2\" (UniqueName: \"kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2\") pod \"payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.674817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.674179    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.674817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.674241    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.682731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.675670    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.781192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.766420    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgp2\" (UniqueName: \"kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2\") pod \"payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.788723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.785566    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:44.788723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.787566    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.788723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.787763    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltp2\" (UniqueName: \"kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2\") pod \"payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.788723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.788321    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"]
Oct 30 21:39:44.789023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.788966    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.790756 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:44.790718    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06858e92_ee3b_4439_8b97_b602de0bbbb6.slice/crio-7f3a49d6e5b7eb926adabfa057c49e6ea162c987d01e4e2c7dacb30d7583ef09 WatchSource:0}: Error finding container 7f3a49d6e5b7eb926adabfa057c49e6ea162c987d01e4e2c7dacb30d7583ef09: Status 404 returned error can't find the container with id 7f3a49d6e5b7eb926adabfa057c49e6ea162c987d01e4e2c7dacb30d7583ef09
Oct 30 21:39:44.807090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.807037    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltp2\" (UniqueName: \"kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2\") pod \"payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.874794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.874747    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:44.923644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:44.910775    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx" event={"ID":"06858e92-ee3b-4439-8b97-b602de0bbbb6","Type":"ContainerStarted","Data":"7f3a49d6e5b7eb926adabfa057c49e6ea162c987d01e4e2c7dacb30d7583ef09"}
Oct 30 21:39:45.142166 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:45.142115    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8fafcf_bb27_46a5_af58_bac88561bf62.slice/crio-935fb27b8ec2e9f07dab9e4312012c20c416bd9409e8a2e0f308cff5ec6853a3 WatchSource:0}: Error finding container 935fb27b8ec2e9f07dab9e4312012c20c416bd9409e8a2e0f308cff5ec6853a3: Status 404 returned error can't find the container with id 935fb27b8ec2e9f07dab9e4312012c20c416bd9409e8a2e0f308cff5ec6853a3
Oct 30 21:39:45.145443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.145364    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"]
Oct 30 21:39:45.238177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.238111    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"]
Oct 30 21:39:45.252703 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:45.252594    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e0ebfe_5a5b_42d7_ae03_7cc476ef3887.slice/crio-8fd822efb1ed64901ff87a7d118490b44fe089d478768c1e6f62c0f6afbe818f WatchSource:0}: Error finding container 8fd822efb1ed64901ff87a7d118490b44fe089d478768c1e6f62c0f6afbe818f: Status 404 returned error can't find the container with id 8fd822efb1ed64901ff87a7d118490b44fe089d478768c1e6f62c0f6afbe818f
Oct 30 21:39:45.948533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.948483    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j" event={"ID":"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887","Type":"ContainerStarted","Data":"2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751"}
Oct 30 21:39:45.948770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.948565    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:45.948770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.948585    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j" event={"ID":"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887","Type":"ContainerStarted","Data":"8fd822efb1ed64901ff87a7d118490b44fe089d478768c1e6f62c0f6afbe818f"}
Oct 30 21:39:45.950895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.950853    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx" event={"ID":"06858e92-ee3b-4439-8b97-b602de0bbbb6","Type":"ContainerStarted","Data":"15e5d22b397b9ae800a04dbaf8e2f8c9bf4df3c4f4f200f24c5d329c7b461578"}
Oct 30 21:39:45.952848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.952812    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7" event={"ID":"cb8fafcf-bb27-46a5-af58-bac88561bf62","Type":"ContainerStarted","Data":"c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b"}
Oct 30 21:39:45.953036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.952878    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7" event={"ID":"cb8fafcf-bb27-46a5-af58-bac88561bf62","Type":"ContainerStarted","Data":"935fb27b8ec2e9f07dab9e4312012c20c416bd9409e8a2e0f308cff5ec6853a3"}
Oct 30 21:39:45.953036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.952909    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:45.996990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:45.995758    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j" podStartSLOduration=1.995733151 podStartE2EDuration="1.995733151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:45.989091839 +0000 UTC m=+1768.911550137" watchObservedRunningTime="2026-10-30 21:39:45.995733151 +0000 UTC m=+1768.918191451"
Oct 30 21:39:46.027452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:46.027402    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx" podStartSLOduration=3.027387763 podStartE2EDuration="3.027387763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:46.026193417 +0000 UTC m=+1768.948651716" watchObservedRunningTime="2026-10-30 21:39:46.027387763 +0000 UTC m=+1768.949846086"
Oct 30 21:39:46.067314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:46.067251    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7" podStartSLOduration=2.067237561 podStartE2EDuration="2.067237561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:46.066111098 +0000 UTC m=+1768.988569396" watchObservedRunningTime="2026-10-30 21:39:46.067237561 +0000 UTC m=+1768.989695858"
Oct 30 21:39:46.982003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:46.981947    6695 generic.go:356] "Generic (PLEG): container finished" podID="06858e92-ee3b-4439-8b97-b602de0bbbb6" containerID="15e5d22b397b9ae800a04dbaf8e2f8c9bf4df3c4f4f200f24c5d329c7b461578" exitCode=0
Oct 30 21:39:46.983671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:46.983631    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx" event={"ID":"06858e92-ee3b-4439-8b97-b602de0bbbb6","Type":"ContainerDied","Data":"15e5d22b397b9ae800a04dbaf8e2f8c9bf4df3c4f4f200f24c5d329c7b461578"}
Oct 30 21:39:47.241537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.241438    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"]
Oct 30 21:39:47.247230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.247198    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.264426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.264384    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"]
Oct 30 21:39:47.322591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.322549    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cl4\" (UniqueName: \"kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.322591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.322592    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.322909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.322754    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.424242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.424202    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.424484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.424444    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cl4\" (UniqueName: \"kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.424557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.424482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.425127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.425098    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.428467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.428387    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.445124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.445087    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cl4\" (UniqueName: \"kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4\") pod \"maas-api-revoke-sub-5aa13844-gz4cj\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.563810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.563770    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:47.897948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:47.897856    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"]
Oct 30 21:39:48.017586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.017510    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj" event={"ID":"73fbb120-e52a-4299-b69b-93552ca361a1","Type":"ContainerStarted","Data":"abed3f359bb5414a6c16a8aaa14a20cd3458b353cfe13519e60a704a34479e8d"}
Oct 30 21:39:48.215135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.215095    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:48.352935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.352888    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca\") pod \"06858e92-ee3b-4439-8b97-b602de0bbbb6\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") "
Oct 30 21:39:48.353156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.352983    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca\") pod \"06858e92-ee3b-4439-8b97-b602de0bbbb6\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") "
Oct 30 21:39:48.353257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.353239    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f\" (UniqueName: \"kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f\") pod \"06858e92-ee3b-4439-8b97-b602de0bbbb6\" (UID: \"06858e92-ee3b-4439-8b97-b602de0bbbb6\") "
Oct 30 21:39:48.353369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.353334    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca" pod "06858e92-ee3b-4439-8b97-b602de0bbbb6" (UID: "06858e92-ee3b-4439-8b97-b602de0bbbb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:48.353891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.353861    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:48.411267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.411219    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f" pod "06858e92-ee3b-4439-8b97-b602de0bbbb6" (UID: "06858e92-ee3b-4439-8b97-b602de0bbbb6"). InnerVolumeSpecName "kube-api-access-h628f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:48.411267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.411241    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca" pod "06858e92-ee3b-4439-8b97-b602de0bbbb6" (UID: "06858e92-ee3b-4439-8b97-b602de0bbbb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:48.455153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.455044    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06858e92-ee3b-4439-8b97-b602de0bbbb6-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:48.455153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:48.455090    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h628f\" (UniqueName: \"kubernetes.io/projected/06858e92-ee3b-4439-8b97-b602de0bbbb6-kube-api-access-h628f\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:49.029809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.029778    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"
Oct 30 21:39:49.029809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.029794    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx" event={"ID":"06858e92-ee3b-4439-8b97-b602de0bbbb6","Type":"ContainerDied","Data":"7f3a49d6e5b7eb926adabfa057c49e6ea162c987d01e4e2c7dacb30d7583ef09"}
Oct 30 21:39:49.030394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.029854    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3a49d6e5b7eb926adabfa057c49e6ea162c987d01e4e2c7dacb30d7583ef09"
Oct 30 21:39:49.032332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.032294    6695 generic.go:356] "Generic (PLEG): container finished" podID="73fbb120-e52a-4299-b69b-93552ca361a1" containerID="2dd4de58d5fca76b9a84631be7fc90f2a09647dad384c7a2a4e99f4192395e84" exitCode=0
Oct 30 21:39:49.032511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.032373    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj" event={"ID":"73fbb120-e52a-4299-b69b-93552ca361a1","Type":"ContainerDied","Data":"2dd4de58d5fca76b9a84631be7fc90f2a09647dad384c7a2a4e99f4192395e84"}
Oct 30 21:39:49.444824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.444775    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:49.445035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.444865    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:49.760485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.760361    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"]
Oct 30 21:39:49.761270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.761242    6695 cpu_manager.go:414] "containerMap: removing container" podUID="06858e92-ee3b-4439-8b97-b602de0bbbb6" containerName="revoke-keys"
Oct 30 21:39:49.761270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.761271    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06858e92-ee3b-4439-8b97-b602de0bbbb6" containerName="revoke-keys"
Oct 30 21:39:49.761517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.761398    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="06858e92-ee3b-4439-8b97-b602de0bbbb6" containerName="revoke-keys"
Oct 30 21:39:49.771838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.771786    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.781371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.781330    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"]
Oct 30 21:39:49.874982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.874937    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.875205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.875112    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.875317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.875289    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5gt\" (UniqueName: \"kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.976586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.976541    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.976813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.976719    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.976893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.976872    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5gt\" (UniqueName: \"kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.977397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.977366    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.979476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.979433    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:49.996110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:49.996053    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5gt\" (UniqueName: \"kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt\") pod \"maas-api-revoke-sub-aa73ef45-5pfsm\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:50.103182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.103138    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:50.288390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.288357    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:50.382852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.382735    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca\") pod \"73fbb120-e52a-4299-b69b-93552ca361a1\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") "
Oct 30 21:39:50.384701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.384656    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca\") pod \"73fbb120-e52a-4299-b69b-93552ca361a1\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") "
Oct 30 21:39:50.385165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.385128    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4\" (UniqueName: \"kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4\") pod \"73fbb120-e52a-4299-b69b-93552ca361a1\" (UID: \"73fbb120-e52a-4299-b69b-93552ca361a1\") "
Oct 30 21:39:50.385747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.385711    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca" pod "73fbb120-e52a-4299-b69b-93552ca361a1" (UID: "73fbb120-e52a-4299-b69b-93552ca361a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:50.386290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.386253    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:50.388234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.388190    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca" pod "73fbb120-e52a-4299-b69b-93552ca361a1" (UID: "73fbb120-e52a-4299-b69b-93552ca361a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:50.388555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.388525    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4" pod "73fbb120-e52a-4299-b69b-93552ca361a1" (UID: "73fbb120-e52a-4299-b69b-93552ca361a1"). InnerVolumeSpecName "kube-api-access-n4cl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:50.436735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.436693    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"]
Oct 30 21:39:50.440777 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:39:50.440735    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43921979_2dac_461b_a941_10f700bb797c.slice/crio-dd2e50a8fd77a3e99fee54c8d548c8261674e6f641281656ccfcc681a2a5b4f3 WatchSource:0}: Error finding container dd2e50a8fd77a3e99fee54c8d548c8261674e6f641281656ccfcc681a2a5b4f3: Status 404 returned error can't find the container with id dd2e50a8fd77a3e99fee54c8d548c8261674e6f641281656ccfcc681a2a5b4f3
Oct 30 21:39:50.487368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.487329    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73fbb120-e52a-4299-b69b-93552ca361a1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:50.487368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:50.487370    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cl4\" (UniqueName: \"kubernetes.io/projected/73fbb120-e52a-4299-b69b-93552ca361a1-kube-api-access-n4cl4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:51.062585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.062537    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm" event={"ID":"43921979-2dac-461b-a941-10f700bb797c","Type":"ContainerStarted","Data":"da85deb03831ce360548e6a007928771ec5cc3cda96e24b636600d83b3fca253"}
Oct 30 21:39:51.062839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.062621    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm" event={"ID":"43921979-2dac-461b-a941-10f700bb797c","Type":"ContainerStarted","Data":"dd2e50a8fd77a3e99fee54c8d548c8261674e6f641281656ccfcc681a2a5b4f3"}
Oct 30 21:39:51.066685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.065955    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"
Oct 30 21:39:51.066685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.065963    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj" event={"ID":"73fbb120-e52a-4299-b69b-93552ca361a1","Type":"ContainerDied","Data":"abed3f359bb5414a6c16a8aaa14a20cd3458b353cfe13519e60a704a34479e8d"}
Oct 30 21:39:51.066685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.066140    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abed3f359bb5414a6c16a8aaa14a20cd3458b353cfe13519e60a704a34479e8d"
Oct 30 21:39:51.070019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.069976    6695 generic.go:356] "Generic (PLEG): container finished" podID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerID="cce57ec403c035183799d0b9483f0a91143fbe077056811bedd8f2ee99e3090d" exitCode=0
Oct 30 21:39:51.070181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.070149    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" event={"ID":"f10ff2df-7e2d-4273-b248-250f6f73d33e","Type":"ContainerDied","Data":"cce57ec403c035183799d0b9483f0a91143fbe077056811bedd8f2ee99e3090d"}
Oct 30 21:39:51.070795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.070772    6695 scope.go:122] "RemoveContainer" containerID="cce57ec403c035183799d0b9483f0a91143fbe077056811bedd8f2ee99e3090d"
Oct 30 21:39:51.073084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.073042    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:51.073204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.073151    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:51.073246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.073200    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:51.104096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:51.104037    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm" podStartSLOduration=2.104020515 podStartE2EDuration="2.104020515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:51.10088068 +0000 UTC m=+1774.023338988" watchObservedRunningTime="2026-10-30 21:39:51.104020515 +0000 UTC m=+1774.026478813"
Oct 30 21:39:52.081238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:52.081201    6695 generic.go:356] "Generic (PLEG): container finished" podID="43921979-2dac-461b-a941-10f700bb797c" containerID="da85deb03831ce360548e6a007928771ec5cc3cda96e24b636600d83b3fca253" exitCode=0
Oct 30 21:39:52.081466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:52.081334    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm" event={"ID":"43921979-2dac-461b-a941-10f700bb797c","Type":"ContainerDied","Data":"da85deb03831ce360548e6a007928771ec5cc3cda96e24b636600d83b3fca253"}
Oct 30 21:39:52.084301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:52.084263    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" event={"ID":"f10ff2df-7e2d-4273-b248-250f6f73d33e","Type":"ContainerStarted","Data":"e12294f73bafb48b38deb1cf468166714d14ddd9d53073b7a5c5306df0241ffc"}
Oct 30 21:39:52.951694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:52.951651    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:52.952233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:52.951961    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:52.952233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:52.952046    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:53.297220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.297192    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:53.356882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.356829    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt\" (UniqueName: \"kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt\") pod \"43921979-2dac-461b-a941-10f700bb797c\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") "
Oct 30 21:39:53.357100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.356933    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca\") pod \"43921979-2dac-461b-a941-10f700bb797c\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") "
Oct 30 21:39:53.357194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.357102    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca\") pod \"43921979-2dac-461b-a941-10f700bb797c\" (UID: \"43921979-2dac-461b-a941-10f700bb797c\") "
Oct 30 21:39:53.357373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.357344    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca" pod "43921979-2dac-461b-a941-10f700bb797c" (UID: "43921979-2dac-461b-a941-10f700bb797c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:53.357643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.357621    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43921979-2dac-461b-a941-10f700bb797c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:53.359199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.359170    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca" pod "43921979-2dac-461b-a941-10f700bb797c" (UID: "43921979-2dac-461b-a941-10f700bb797c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:53.359326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.359217    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt" pod "43921979-2dac-461b-a941-10f700bb797c" (UID: "43921979-2dac-461b-a941-10f700bb797c"). InnerVolumeSpecName "kube-api-access-dh5gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:53.459400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.459344    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43921979-2dac-461b-a941-10f700bb797c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:53.459400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.459389    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5gt\" (UniqueName: \"kubernetes.io/projected/43921979-2dac-461b-a941-10f700bb797c-kube-api-access-dh5gt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:39:53.952339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.952297    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:53.952850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:53.952360    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:54.109872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:54.109839    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"
Oct 30 21:39:54.109872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:54.109858    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm" event={"ID":"43921979-2dac-461b-a941-10f700bb797c","Type":"ContainerDied","Data":"dd2e50a8fd77a3e99fee54c8d548c8261674e6f641281656ccfcc681a2a5b4f3"}
Oct 30 21:39:54.110117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:54.109916    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd2e50a8fd77a3e99fee54c8d548c8261674e6f641281656ccfcc681a2a5b4f3"
Oct 30 21:39:54.974629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:54.968168    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:54.974629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:54.968264    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.530734    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"]
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.531636    6695 cpu_manager.go:414] "containerMap: removing container" podUID="73fbb120-e52a-4299-b69b-93552ca361a1" containerName="revoke-keys"
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.531659    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fbb120-e52a-4299-b69b-93552ca361a1" containerName="revoke-keys"
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.531693    6695 cpu_manager.go:414] "containerMap: removing container" podUID="43921979-2dac-461b-a941-10f700bb797c" containerName="revoke-keys"
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.531702    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43921979-2dac-461b-a941-10f700bb797c" containerName="revoke-keys"
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.531905    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fbb120-e52a-4299-b69b-93552ca361a1" containerName="revoke-keys"
Oct 30 21:39:55.535815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.531924    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="43921979-2dac-461b-a941-10f700bb797c" containerName="revoke-keys"
Oct 30 21:39:55.539474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.539441    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.545819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.545783    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"]
Oct 30 21:39:55.554412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.554363    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:55.565202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.565164    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:55.565202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.565164    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:55.565495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.565299    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:39:55.593933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.593884    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl92t\" (UniqueName: \"kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.594140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.594044    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.594190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.594157    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.695583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.695542    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.695841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.695717    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl92t\" (UniqueName: \"kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.695841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.695796    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.696361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.696326    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.698272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.698240    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.767624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.763054    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl92t\" (UniqueName: \"kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t\") pod \"maas-api-revoke-sub-b3061e6b-jqwzt\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:55.883695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:55.880058    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:39:56.030109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:56.029230    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:56.030109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:56.029291    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:56.732432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:56.731017    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"]
Oct 30 21:39:56.962670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:56.962143    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:56.962670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:56.962212    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:57.000698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:57.000582    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:39:57.001249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:57.001222    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:39:57.849629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:57.847740    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt" event={"ID":"672d4db1-6b2e-400f-94f3-29e24b477885","Type":"ContainerStarted","Data":"22474d099ab1c6addf9831428a287f733b2b4d9df1cb212171a8338627f1b461"}
Oct 30 21:39:57.954183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:57.954134    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:57.954541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:57.954505    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:58.512631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.502729    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:58.614659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.604793    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:58.614659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.605182    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:39:58.614659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.605441    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:39:58.845802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.845731    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:58.894658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.894489    6695 generic.go:356] "Generic (PLEG): container finished" podID="672d4db1-6b2e-400f-94f3-29e24b477885" containerID="03f4af3f2b2aec74d76f15155dc1f8a309da58f30db05bee2edfa340f21e9782" exitCode=0
Oct 30 21:39:58.894658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.894573    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt" event={"ID":"672d4db1-6b2e-400f-94f3-29e24b477885","Type":"ContainerDied","Data":"03f4af3f2b2aec74d76f15155dc1f8a309da58f30db05bee2edfa340f21e9782"}
Oct 30 21:39:58.962810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.962764    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:39:58.963123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:58.963091    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Oct 30 21:39:59.931678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:59.931584    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"]
Oct 30 21:39:59.935785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:59.935745    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:39:59.955133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:59.954848    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"]
Oct 30 21:39:59.960394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:59.960355    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:59.960930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:59.960908    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:39:59.963507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:39:59.963481    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:40:00.100353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.100314    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.100562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.100531    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxfj\" (UniqueName: \"kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.100753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.100735    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.114198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.114151    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"]
Oct 30 21:40:00.118860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.118833    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.121722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.121693    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:40:00.144077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.142291    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"]
Oct 30 21:40:00.202655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.202523    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca\") pod \"672d4db1-6b2e-400f-94f3-29e24b477885\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") "
Oct 30 21:40:00.202655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.202621    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca\") pod \"672d4db1-6b2e-400f-94f3-29e24b477885\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") "
Oct 30 21:40:00.202914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.202854    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t\" (UniqueName: \"kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t\") pod \"672d4db1-6b2e-400f-94f3-29e24b477885\" (UID: \"672d4db1-6b2e-400f-94f3-29e24b477885\") "
Oct 30 21:40:00.203095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203066    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxfj\" (UniqueName: \"kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.203186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203163    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.203327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203296    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7fz\" (UniqueName: \"kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.203405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203335    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.203405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203335    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca" pod "672d4db1-6b2e-400f-94f3-29e24b477885" (UID: "672d4db1-6b2e-400f-94f3-29e24b477885"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:00.203580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203555    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.203756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203735    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.203864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.203845    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:00.204085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.204062    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.205494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.205459    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca" pod "672d4db1-6b2e-400f-94f3-29e24b477885" (UID: "672d4db1-6b2e-400f-94f3-29e24b477885"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:00.205494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.205481    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t" pod "672d4db1-6b2e-400f-94f3-29e24b477885" (UID: "672d4db1-6b2e-400f-94f3-29e24b477885"). InnerVolumeSpecName "kube-api-access-gl92t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:00.206404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.206377    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.224051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.224010    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxfj\" (UniqueName: \"kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj\") pod \"maas-api-revoke-sub-9cd644d0-z9h4g\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.256817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.256772    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:00.313667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.305066    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7fz\" (UniqueName: \"kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.313667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.305199    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.313667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.310491    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.313667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.310745    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.313667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.310938    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl92t\" (UniqueName: \"kubernetes.io/projected/672d4db1-6b2e-400f-94f3-29e24b477885-kube-api-access-gl92t\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:00.313667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.310960    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672d4db1-6b2e-400f-94f3-29e24b477885-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:00.316369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.314725    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.330700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.330664    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7fz\" (UniqueName: \"kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz\") pod \"maas-api-revoke-sub-dc68434d-49bw8\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.446434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.446387    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:00.613819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.613768    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"]
Oct 30 21:40:00.629320 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:00.629280    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab98276_242e_4bae_a6dc_4114927e5930.slice/crio-ac992bf5c02df62ede10507883073312d60c4726a507c910c9abbaffa0613cc0 WatchSource:0}: Error finding container ac992bf5c02df62ede10507883073312d60c4726a507c910c9abbaffa0613cc0: Status 404 returned error can't find the container with id ac992bf5c02df62ede10507883073312d60c4726a507c910c9abbaffa0613cc0
Oct 30 21:40:00.798951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.798909    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"]
Oct 30 21:40:00.813475 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:00.813429    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb721028d_8260_4a62_90f4_06d45e308405.slice/crio-dbcd94e5d735a80849d3dd56fe84c0fc664b884df66f2370feb99c81d84f9c6b WatchSource:0}: Error finding container dbcd94e5d735a80849d3dd56fe84c0fc664b884df66f2370feb99c81d84f9c6b: Status 404 returned error can't find the container with id dbcd94e5d735a80849d3dd56fe84c0fc664b884df66f2370feb99c81d84f9c6b
Oct 30 21:40:00.837430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.837391    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"]
Oct 30 21:40:00.838418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.838393    6695 cpu_manager.go:414] "containerMap: removing container" podUID="672d4db1-6b2e-400f-94f3-29e24b477885" containerName="revoke-keys"
Oct 30 21:40:00.838577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.838421    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672d4db1-6b2e-400f-94f3-29e24b477885" containerName="revoke-keys"
Oct 30 21:40:00.838683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.838653    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="672d4db1-6b2e-400f-94f3-29e24b477885" containerName="revoke-keys"
Oct 30 21:40:00.843666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.843582    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:00.861890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.861848    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"]
Oct 30 21:40:00.914474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.914428    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8" event={"ID":"b721028d-8260-4a62-90f4-06d45e308405","Type":"ContainerStarted","Data":"dbcd94e5d735a80849d3dd56fe84c0fc664b884df66f2370feb99c81d84f9c6b"}
Oct 30 21:40:00.916458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.916411    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g" event={"ID":"3ab98276-242e-4bae-a6dc-4114927e5930","Type":"ContainerStarted","Data":"ac992bf5c02df62ede10507883073312d60c4726a507c910c9abbaffa0613cc0"}
Oct 30 21:40:00.918359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.918324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt" event={"ID":"672d4db1-6b2e-400f-94f3-29e24b477885","Type":"ContainerDied","Data":"22474d099ab1c6addf9831428a287f733b2b4d9df1cb212171a8338627f1b461"}
Oct 30 21:40:00.918359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.918358    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"
Oct 30 21:40:00.918578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.918385    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22474d099ab1c6addf9831428a287f733b2b4d9df1cb212171a8338627f1b461"
Oct 30 21:40:00.927953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.927911    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:00.928118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.928066    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:00.928202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:00.928152    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rh6z\" (UniqueName: \"kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.029669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.029631    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.030143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.029800    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.030143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.029912    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rh6z\" (UniqueName: \"kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.030516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.030488    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.032757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.032724    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.051094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.051015    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rh6z\" (UniqueName: \"kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z\") pod \"maas-api-revoke-sub-94e57cc4-ck9zh\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.077744 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:40:01.077707    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:40:01.159850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.159807    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:01.480754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.480494    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"]
Oct 30 21:40:01.484472 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:01.484437    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadfcda4_a93d_4c3e_9e1e_d6d1a866c7ee.slice/crio-fd8f056dde576cf6032b1d303ea70a4e2aab65735143c4b3857fc98e3503a405 WatchSource:0}: Error finding container fd8f056dde576cf6032b1d303ea70a4e2aab65735143c4b3857fc98e3503a405: Status 404 returned error can't find the container with id fd8f056dde576cf6032b1d303ea70a4e2aab65735143c4b3857fc98e3503a405
Oct 30 21:40:01.931121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.931080    6695 generic.go:356] "Generic (PLEG): container finished" podID="b721028d-8260-4a62-90f4-06d45e308405" containerID="ce158dfae0858d930cbacbc3d5bd6cec023138c5f4fb3ac99f78df043b956e1f" exitCode=0
Oct 30 21:40:01.931313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.931211    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8" event={"ID":"b721028d-8260-4a62-90f4-06d45e308405","Type":"ContainerDied","Data":"ce158dfae0858d930cbacbc3d5bd6cec023138c5f4fb3ac99f78df043b956e1f"}
Oct 30 21:40:01.933436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.933399    6695 generic.go:356] "Generic (PLEG): container finished" podID="3ab98276-242e-4bae-a6dc-4114927e5930" containerID="0f3dbaece1617d5a98b0935a23f2beb9927f1cf96cf944ea6cad4d967715d3b9" exitCode=0
Oct 30 21:40:01.933690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.933506    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g" event={"ID":"3ab98276-242e-4bae-a6dc-4114927e5930","Type":"ContainerDied","Data":"0f3dbaece1617d5a98b0935a23f2beb9927f1cf96cf944ea6cad4d967715d3b9"}
Oct 30 21:40:01.935709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:01.935669    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh" event={"ID":"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee","Type":"ContainerStarted","Data":"fd8f056dde576cf6032b1d303ea70a4e2aab65735143c4b3857fc98e3503a405"}
Oct 30 21:40:02.947299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:02.947195    6695 generic.go:356] "Generic (PLEG): container finished" podID="badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" containerID="3f94f78eb588d0043f147a34b8600f1de93d2adc78890062dcce2348f969b0fe" exitCode=0
Oct 30 21:40:02.947838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:02.947287    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh" event={"ID":"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee","Type":"ContainerDied","Data":"3f94f78eb588d0043f147a34b8600f1de93d2adc78890062dcce2348f969b0fe"}
Oct 30 21:40:02.960257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:02.960211    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"]
Oct 30 21:40:02.965365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:02.965327    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.005064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.004948    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"]
Oct 30 21:40:03.067312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067254    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.067520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067492    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.067652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067617    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.067742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067680    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.067811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067780    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.067865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067827    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.068009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.067986    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsxg\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.068146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.068117    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.068302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.068171    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.169967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.169932    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170011    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170091    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170130    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170172    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170216    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170320    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsxg\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170425    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170697    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170739    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170863    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.170944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.170598    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.171444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.171420    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.172923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.172890    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.173042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.173020    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.189327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.189292    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.189616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.189582    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsxg\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.218068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.217991    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:03.220102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.220077    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:03.287587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.287538    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:03.373419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.373371    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj\" (UniqueName: \"kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj\") pod \"3ab98276-242e-4bae-a6dc-4114927e5930\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") "
Oct 30 21:40:03.373676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.373430    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca\") pod \"b721028d-8260-4a62-90f4-06d45e308405\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") "
Oct 30 21:40:03.373676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.373529    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca\") pod \"3ab98276-242e-4bae-a6dc-4114927e5930\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") "
Oct 30 21:40:03.373676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.373581    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz\" (UniqueName: \"kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz\") pod \"b721028d-8260-4a62-90f4-06d45e308405\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") "
Oct 30 21:40:03.373950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.373693    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca\") pod \"3ab98276-242e-4bae-a6dc-4114927e5930\" (UID: \"3ab98276-242e-4bae-a6dc-4114927e5930\") "
Oct 30 21:40:03.373950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.373824    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca\") pod \"b721028d-8260-4a62-90f4-06d45e308405\" (UID: \"b721028d-8260-4a62-90f4-06d45e308405\") "
Oct 30 21:40:03.375976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.375935    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca" pod "b721028d-8260-4a62-90f4-06d45e308405" (UID: "b721028d-8260-4a62-90f4-06d45e308405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:03.376165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.376144    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca" pod "3ab98276-242e-4bae-a6dc-4114927e5930" (UID: "3ab98276-242e-4bae-a6dc-4114927e5930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:03.378179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.378134    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj" pod "3ab98276-242e-4bae-a6dc-4114927e5930" (UID: "3ab98276-242e-4bae-a6dc-4114927e5930"). InnerVolumeSpecName "kube-api-access-8hxfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:03.378346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.378322    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz" pod "b721028d-8260-4a62-90f4-06d45e308405" (UID: "b721028d-8260-4a62-90f4-06d45e308405"). InnerVolumeSpecName "kube-api-access-lh7fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:03.379484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.379453    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca" pod "3ab98276-242e-4bae-a6dc-4114927e5930" (UID: "3ab98276-242e-4bae-a6dc-4114927e5930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:03.380660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.380623    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca" pod "b721028d-8260-4a62-90f4-06d45e308405" (UID: "b721028d-8260-4a62-90f4-06d45e308405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:03.475801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.475706    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7fz\" (UniqueName: \"kubernetes.io/projected/b721028d-8260-4a62-90f4-06d45e308405-kube-api-access-lh7fz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:03.475801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.475740    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:03.475801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.475751    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b721028d-8260-4a62-90f4-06d45e308405-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:03.475801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.475760    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hxfj\" (UniqueName: \"kubernetes.io/projected/3ab98276-242e-4bae-a6dc-4114927e5930-kube-api-access-8hxfj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:03.475801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.475770    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b721028d-8260-4a62-90f4-06d45e308405-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:03.475801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.475779    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab98276-242e-4bae-a6dc-4114927e5930-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:03.622070 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.622036    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"]
Oct 30 21:40:03.625526 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:03.625482    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61919737_bb02_4132_8865_c1b33a99b398.slice/crio-ca78c92fa6bb4246d3c14d529869584b22adbe38090b3b2f8c70be175851d075 WatchSource:0}: Error finding container ca78c92fa6bb4246d3c14d529869584b22adbe38090b3b2f8c70be175851d075: Status 404 returned error can't find the container with id ca78c92fa6bb4246d3c14d529869584b22adbe38090b3b2f8c70be175851d075
Oct 30 21:40:03.628523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.628483    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:03.628666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.628564    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:03.628666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.628596    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:03.959428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.959387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" event={"ID":"61919737-bb02-4132-8865-c1b33a99b398","Type":"ContainerStarted","Data":"a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350"}
Oct 30 21:40:03.959967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.959453    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" event={"ID":"61919737-bb02-4132-8865-c1b33a99b398","Type":"ContainerStarted","Data":"ca78c92fa6bb4246d3c14d529869584b22adbe38090b3b2f8c70be175851d075"}
Oct 30 21:40:03.961342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.961315    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"
Oct 30 21:40:03.961495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.961341    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8" event={"ID":"b721028d-8260-4a62-90f4-06d45e308405","Type":"ContainerDied","Data":"dbcd94e5d735a80849d3dd56fe84c0fc664b884df66f2370feb99c81d84f9c6b"}
Oct 30 21:40:03.961495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.961393    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbcd94e5d735a80849d3dd56fe84c0fc664b884df66f2370feb99c81d84f9c6b"
Oct 30 21:40:03.963097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.963064    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"
Oct 30 21:40:03.963253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.963097    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g" event={"ID":"3ab98276-242e-4bae-a6dc-4114927e5930","Type":"ContainerDied","Data":"ac992bf5c02df62ede10507883073312d60c4726a507c910c9abbaffa0613cc0"}
Oct 30 21:40:03.963253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:03.963149    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac992bf5c02df62ede10507883073312d60c4726a507c910c9abbaffa0613cc0"
Oct 30 21:40:04.005674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.005313    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podStartSLOduration=2.005294686 podStartE2EDuration="2.005294686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:04.000794715 +0000 UTC m=+1786.923253012" watchObservedRunningTime="2026-10-30 21:40:04.005294686 +0000 UTC m=+1786.927752986"
Oct 30 21:40:04.113848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.113815    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:04.200248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.200202    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca\") pod \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") "
Oct 30 21:40:04.200468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.200323    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca\") pod \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") "
Oct 30 21:40:04.200468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.200447    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z\" (UniqueName: \"kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z\") pod \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\" (UID: \"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee\") "
Oct 30 21:40:04.200745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.200712    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca" pod "badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" (UID: "badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:04.201107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.201077    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:04.208272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.208223    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca" pod "badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" (UID: "badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:04.208596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.208256    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z" pod "badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" (UID: "badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee"). InnerVolumeSpecName "kube-api-access-4rh6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:04.287836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.287782    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:04.290032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.289992    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:04.290191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.290060    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:04.302792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.302757    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:04.302792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.302789    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rh6z\" (UniqueName: \"kubernetes.io/projected/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee-kube-api-access-4rh6z\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:04.972908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.972852    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh" event={"ID":"badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee","Type":"ContainerDied","Data":"fd8f056dde576cf6032b1d303ea70a4e2aab65735143c4b3857fc98e3503a405"}
Oct 30 21:40:04.972908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.972898    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"
Oct 30 21:40:04.973390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:04.972967    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd8f056dde576cf6032b1d303ea70a4e2aab65735143c4b3857fc98e3503a405"
Oct 30 21:40:05.289014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:05.288976    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:05.289176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:05.289035    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:06.288810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:06.288765    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:06.289284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:06.288822    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:07.288247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.288210    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:07.288437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.288261    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:07.796411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.796364    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"]
Oct 30 21:40:07.797117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797088    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3ab98276-242e-4bae-a6dc-4114927e5930" containerName="revoke-keys"
Oct 30 21:40:07.797117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797114    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab98276-242e-4bae-a6dc-4114927e5930" containerName="revoke-keys"
Oct 30 21:40:07.797325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797136    6695 cpu_manager.go:414] "containerMap: removing container" podUID="badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" containerName="revoke-keys"
Oct 30 21:40:07.797325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797142    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" containerName="revoke-keys"
Oct 30 21:40:07.797325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797158    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b721028d-8260-4a62-90f4-06d45e308405" containerName="revoke-keys"
Oct 30 21:40:07.797325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797164    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b721028d-8260-4a62-90f4-06d45e308405" containerName="revoke-keys"
Oct 30 21:40:07.797325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797316    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab98276-242e-4bae-a6dc-4114927e5930" containerName="revoke-keys"
Oct 30 21:40:07.797560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797333    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" containerName="revoke-keys"
Oct 30 21:40:07.797560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.797346    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b721028d-8260-4a62-90f4-06d45e308405" containerName="revoke-keys"
Oct 30 21:40:07.803808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.803784    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.844130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.838890    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-09eabe-openshift-default-dockercfg-dqwhz\""
Oct 30 21:40:07.853699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.850720    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"]
Oct 30 21:40:07.882479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882443    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882572    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882615    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882652    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75gm\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882678    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882706    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882758    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882793    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.882852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.882816    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.983983    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984107    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984288    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75gm\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984349    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984394    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984441    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984470    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984472    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984549    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984584    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984741    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.984787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.984761    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.985211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.985190    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.986739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.986709    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:07.986839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:07.986819    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:08.040711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.037324    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:08.049666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.048291    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75gm\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:08.169626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.162858    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:08.288379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.288343    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:08.288535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.288414    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:08.564431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.564400    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"]
Oct 30 21:40:08.567806 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:08.567765    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bfd09c_2941_477e_8e75_f06def9a9920.slice/crio-00ede815bcca3f80cffb141123019253a57a273181c28988b1621ee87cf4584a WatchSource:0}: Error finding container 00ede815bcca3f80cffb141123019253a57a273181c28988b1621ee87cf4584a: Status 404 returned error can't find the container with id 00ede815bcca3f80cffb141123019253a57a273181c28988b1621ee87cf4584a
Oct 30 21:40:08.570588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.570536    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:08.570689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.570651    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:08.570733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:08.570693    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:09.035625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.027719    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" event={"ID":"77bfd09c-2941-477e-8e75-f06def9a9920","Type":"ContainerStarted","Data":"d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"}
Oct 30 21:40:09.035625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.027790    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" event={"ID":"77bfd09c-2941-477e-8e75-f06def9a9920","Type":"ContainerStarted","Data":"00ede815bcca3f80cffb141123019253a57a273181c28988b1621ee87cf4584a"}
Oct 30 21:40:09.074303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.074249    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podStartSLOduration=2.074235496 podStartE2EDuration="2.074235496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.073771435 +0000 UTC m=+1791.996229733" watchObservedRunningTime="2026-10-30 21:40:09.074235496 +0000 UTC m=+1791.996693792"
Oct 30 21:40:09.163506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.163470    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:09.168753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.168722    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:09.292664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.292549    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:09.292946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:09.292917    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:10.038134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:10.038105    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:10.038995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:10.038972    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:40:10.288661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:10.288551    6695 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Oct 30 21:40:10.288810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:10.288652    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Oct 30 21:40:11.292522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:11.292493    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:11.293010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:11.292997    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:11.293715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:11.293694    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:40:12.306782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.296285    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"]
Oct 30 21:40:12.313338 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.312540    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.365292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.365256    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"]
Oct 30 21:40:12.370649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.370007    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3172e5-openshift-default-dockercfg-878wj\""
Oct 30 21:40:12.415657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.415616    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.415840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.415766    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.415944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.415921    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.416040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.416022    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg88q\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.416103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.416092    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.416166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.416125    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.416223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.416173    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.416316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.416294    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.416376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.416340    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539031    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539087    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539130    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539165    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539234    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539293    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539398    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539481    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg88q\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.539579    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.591749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.540021    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.608656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.597242    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.608656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.597616    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.608656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.597863    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.608656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.598188    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.638221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.638123    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.652654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.652550    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.670718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.669684    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.672163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.672108    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg88q\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q\") pod \"e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") " pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:12.712622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:12.711783    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:13.269810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.269778    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"]
Oct 30 21:40:13.275647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.275565    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.309134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.309085    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-5494e8-openshift-default-dockercfg-jppjn\""
Oct 30 21:40:13.324498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.322692    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"]
Oct 30 21:40:13.328054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.328018    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"]
Oct 30 21:40:13.345342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.345298    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:13.345581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.345557    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:13.345796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.345755    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.406925    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.406977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407023    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407062    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407115    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407264    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407313    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxhq\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407354    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.416658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.407388    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.511636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.511415    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.511636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.511497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxhq\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.511636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.511545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.511636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.511583    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.536675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.535128    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.536675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.535315    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.536675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.535350    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.536675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.535403    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.536675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.535444    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.536675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.535508    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.559864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.539164    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.559864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.539431    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.559864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.539728    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.559864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.539992    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.559864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.540319    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.602694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.597216    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.602694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.599835    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.621639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.614002    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxhq\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:13.636667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:13.632446    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:14.211316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.211275    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" event={"ID":"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73","Type":"ContainerStarted","Data":"10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"}
Oct 30 21:40:14.211660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.211638    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" event={"ID":"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73","Type":"ContainerStarted","Data":"10b06ae035810c809d544c2e2a105a7cb25709d1b2c3603cabb61b4423f9a453"}
Oct 30 21:40:14.286143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.286047    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podStartSLOduration=2.2860268919999998 podStartE2EDuration="2.286026892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:14.283393472 +0000 UTC m=+1797.205851777" watchObservedRunningTime="2026-10-30 21:40:14.286026892 +0000 UTC m=+1797.208485191"
Oct 30 21:40:14.611830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.611780    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:14.717387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.717356    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:14.850860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.850797    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:14.935849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:14.935764    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:40:15.256683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.244454    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 30 21:40:15.492279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.492223    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:15.635684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.619266    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.075795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.636357    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.120659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.660624    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:40:16.120659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.705062    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:40:16.120659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.866117    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.120659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.931950    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"]
Oct 30 21:40:16.128631 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:15.938185    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c4c2b4_4a4b_41d6_aa81_66f0da709615.slice/crio-fc984bb19ec75e6f13c4cfd7a70f1bec8fa96340a99208fcfc60c7ca2baff310 WatchSource:0}: Error finding container fc984bb19ec75e6f13c4cfd7a70f1bec8fa96340a99208fcfc60c7ca2baff310: Status 404 returned error can't find the container with id fc984bb19ec75e6f13c4cfd7a70f1bec8fa96340a99208fcfc60c7ca2baff310
Oct 30 21:40:16.128631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:15.950443    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.150071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.150024    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:16.150350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.150325    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:16.150495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.150478    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:16.204649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.192570    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:40:16.204649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.192639    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"]
Oct 30 21:40:16.204649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.192706    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:40:16.204649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.192775    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:16.204649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.192820    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:40:16.204649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.193851    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:16.242733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.242700    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-09eabe\""
Oct 30 21:40:16.261145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.256296    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-09eabe-dockercfg-s6bzb\""
Oct 30 21:40:16.384693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.380918    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwz97\" (UniqueName: \"kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97\") pod \"payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:16.478041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.477971    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"]
Oct 30 21:40:16.516139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.516086    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.568183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.568152    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:40:16.575712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.575669    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"]
Oct 30 21:40:16.721412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.708361    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:16.854648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.849063    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 21:40:16.871744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.859745    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:16.942959 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:16.871687    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"]
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012090    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012171    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012242    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012308    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012355    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012390    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwz97\" (UniqueName: \"kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97\") pod \"payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012421    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcm72\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012453    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012668    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:17.012707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012704    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.013431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.012745    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.101678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.076505    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:17.101678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.098336    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:40:17.171720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.105686    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.171720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.105716    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.171720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.129674    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:40:17.171720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.129750    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"]
Oct 30 21:40:17.171720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.136765    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193192    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193272    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193343    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193390    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193465    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193521    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193571    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193627    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcm72\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.193653    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.194579    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.287339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.285194    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.288794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.265125    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.480548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.480505    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.565917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.522137    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.565917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.522443    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.565917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.532977    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.917689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.913296    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:17.933748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.933715    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwz97\" (UniqueName: \"kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97\") pod \"payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:17.943721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.942025    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.944008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.942355    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:17.963720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.961397    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjhj\" (UniqueName: \"kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj\") pod \"payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:17.963720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.961647    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:18.010655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:17.966942    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcm72\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:18.309625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.104994    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:18.314306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.139953    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:40:18.316003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.249999    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:18.316619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.251142    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:40:18.344148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.344114    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:18.426751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.268831    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:18.463333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.251570    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-09eabe-dockercfg-s6bzb\""
Oct 30 21:40:18.679707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.679167    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:18.679707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.679516    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjhj\" (UniqueName: \"kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj\") pod \"payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:19.010462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:18.983755    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:19.080178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.056229    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:19.140649 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:40:19.121325    6695 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.73s"
Oct 30 21:40:19.140649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.140309    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" event={"ID":"e4c4c2b4-4a4b-41d6-aa81-66f0da709615","Type":"ContainerStarted","Data":"fc984bb19ec75e6f13c4cfd7a70f1bec8fa96340a99208fcfc60c7ca2baff310"}
Oct 30 21:40:19.140649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.140352    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"]
Oct 30 21:40:19.203725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.203587    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"]
Oct 30 21:40:19.203725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.203663    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"]
Oct 30 21:40:19.204004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.203739    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:40:19.204004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.203768    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:19.204004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.203816    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:40:19.204004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.203854    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 21:40:19.210031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.206052    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.215944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.215793    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjhj\" (UniqueName: \"kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj\") pod \"payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:19.278715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.278355    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.278715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.278479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.278715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.278658    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.279099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.278726    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.279099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.278849    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.279099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.278940    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.279099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.279002    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.279099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.279072    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.279334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.279119    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk6d\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.294378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.294253    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 21:40:19.294378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.294315    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 21:40:19.294378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.294352    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:40:19.294378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.294385    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.380826    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk6d\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.380919    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381010    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381105    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381152    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381321    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381376    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.381437    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.382795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.382768    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.385592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.385245    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.387555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.386115    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.387555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.386362    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.387555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.386776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.390334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.390141    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.493675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.493622    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.510025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.509992    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:19.586122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.575468    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk6d\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.586122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.575510    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.596644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.596471    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:19.663331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.663290    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" event={"ID":"e4c4c2b4-4a4b-41d6-aa81-66f0da709615","Type":"ContainerStarted","Data":"75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"}
Oct 30 21:40:19.723449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.723354    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:40:19.761283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.761170    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:40:19.786235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.782950    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podStartSLOduration=6.782920287 podStartE2EDuration="6.782920287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.777370712 +0000 UTC m=+1802.699829030" watchObservedRunningTime="2026-10-30 21:40:19.782920287 +0000 UTC m=+1802.705378586"
Oct 30 21:40:19.848963 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:19.848917    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2190db3a_850e_4ca5_a71b_436879f0988a.slice/crio-27b975a09daa9ca334bf3f0a52b7900e8641c29a11d453fa66c184f8731987a6 WatchSource:0}: Error finding container 27b975a09daa9ca334bf3f0a52b7900e8641c29a11d453fa66c184f8731987a6: Status 404 returned error can't find the container with id 27b975a09daa9ca334bf3f0a52b7900e8641c29a11d453fa66c184f8731987a6
Oct 30 21:40:19.851239 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:19.851115    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b52dd2_cbbe_47bb_ab7d_4a259c376588.slice/crio-489281b0d3ec07c1277503a93d850ba8286f0946ff657b2d397a7bf9dc18685b WatchSource:0}: Error finding container 489281b0d3ec07c1277503a93d850ba8286f0946ff657b2d397a7bf9dc18685b: Status 404 returned error can't find the container with id 489281b0d3ec07c1277503a93d850ba8286f0946ff657b2d397a7bf9dc18685b
Oct 30 21:40:19.852044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.852003    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"]
Oct 30 21:40:19.853274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.853235    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:19.853625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.853486    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:19.853831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.853806    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:19.861716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.861678    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"]
Oct 30 21:40:19.867555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.867093    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"]
Oct 30 21:40:19.877382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.877348    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:19.884842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.884795    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"]
Oct 30 21:40:19.893390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.893355    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-3172e5\""
Oct 30 21:40:19.893390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:19.893354    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-3172e5-dockercfg-vdlts\""
Oct 30 21:40:20.266773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.264511    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr676\" (UniqueName: \"kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676\") pod \"payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.266773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.264749    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.274800    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"]
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.285580    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"]
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.285767    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.286058    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.286226    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.286328    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.288533    6695 scope.go:122] "RemoveContainer" containerID="f524b374afa3ca09ce1abf8f9236e6077ce6786860634f1e7545e055eea8c9a2"
Oct 30 21:40:20.292274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.291763    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"]
Oct 30 21:40:20.301844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.301034    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"]
Oct 30 21:40:20.364214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.364179    6695 scope.go:122] "RemoveContainer" containerID="09169b24b4d5add2f65dfc7f9162b6cfadb089282b1b02dc15895cae590d404c"
Oct 30 21:40:20.367995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.365941    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr676\" (UniqueName: \"kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676\") pod \"payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.367995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.366059    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.367995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.366394    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6sc\" (UniqueName: \"kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc\") pod \"payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.367995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.366788    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.367995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.367775    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.387177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.387138    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr676\" (UniqueName: \"kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676\") pod \"payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.435451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.430828    6695 scope.go:122] "RemoveContainer" containerID="f758e19284ef8b4d85735f7aadb2bbb395069e23de3b9d3c9a0849f44bac2a03"
Oct 30 21:40:20.468391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.468332    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.468647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.468575    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6sc\" (UniqueName: \"kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc\") pod \"payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.469912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.469879    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.470849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.470707    6695 scope.go:122] "RemoveContainer" containerID="08a826deb5a844dfaf4ae0d173018a8b6775b165d05e05cd46fa7f8c20338ba9"
Oct 30 21:40:20.488525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.488451    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6sc\" (UniqueName: \"kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc\") pod \"payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.576663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.576237    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:20.645154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.645080    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:20.648549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.648434    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:20.653808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.653032    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:20.700664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.697979    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" event={"ID":"a711ee9f-5969-4295-92a6-c0e6a560a8e4","Type":"ContainerStarted","Data":"54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2"}
Oct 30 21:40:20.700664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.698059    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" event={"ID":"a711ee9f-5969-4295-92a6-c0e6a560a8e4","Type":"ContainerStarted","Data":"01a947e94eff46c1d2fb3f076912785a8908f01c89dbe17dc4cfd92263c77ede"}
Oct 30 21:40:20.700664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.700540    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" event={"ID":"c6194418-0e26-4d4e-bd08-d44266b3ca6e","Type":"ContainerStarted","Data":"02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27"}
Oct 30 21:40:20.700664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.700620    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" event={"ID":"c6194418-0e26-4d4e-bd08-d44266b3ca6e","Type":"ContainerStarted","Data":"0889ad19648a4f4b9dfc213569acd4c4e1587549e20bcbf1b0a89998071d279d"}
Oct 30 21:40:20.701561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.701502    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:20.704704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.704658    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" event={"ID":"e1b52dd2-cbbe-47bb-ab7d-4a259c376588","Type":"ContainerStarted","Data":"99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95"}
Oct 30 21:40:20.704899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.704729    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" event={"ID":"e1b52dd2-cbbe-47bb-ab7d-4a259c376588","Type":"ContainerStarted","Data":"489281b0d3ec07c1277503a93d850ba8286f0946ff657b2d397a7bf9dc18685b"}
Oct 30 21:40:20.707156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.706979    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:20.759874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.709540    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" event={"ID":"2190db3a-850e-4ca5-a71b-436879f0988a","Type":"ContainerStarted","Data":"2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213"}
Oct 30 21:40:20.759874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.709633    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" event={"ID":"2190db3a-850e-4ca5-a71b-436879f0988a","Type":"ContainerStarted","Data":"27b975a09daa9ca334bf3f0a52b7900e8641c29a11d453fa66c184f8731987a6"}
Oct 30 21:40:20.759874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.711008    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:20.759874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.757936    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:40:20.761349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.761285    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podStartSLOduration=5.76126566 podStartE2EDuration="5.76126566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:20.758160257 +0000 UTC m=+1803.680618556" watchObservedRunningTime="2026-10-30 21:40:20.76126566 +0000 UTC m=+1803.683723961"
Oct 30 21:40:20.803718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.802433    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" podStartSLOduration=5.802417651 podStartE2EDuration="5.802417651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:20.79849237 +0000 UTC m=+1803.720950668" watchObservedRunningTime="2026-10-30 21:40:20.802417651 +0000 UTC m=+1803.724875948"
Oct 30 21:40:20.856280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.855808    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" podStartSLOduration=5.855783494 podStartE2EDuration="5.855783494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:20.854880742 +0000 UTC m=+1803.777339061" watchObservedRunningTime="2026-10-30 21:40:20.855783494 +0000 UTC m=+1803.778241789"
Oct 30 21:40:20.964150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.964039    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podStartSLOduration=5.964008131 podStartE2EDuration="5.964008131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:20.913853922 +0000 UTC m=+1803.836312221" watchObservedRunningTime="2026-10-30 21:40:20.964008131 +0000 UTC m=+1803.886466445"
Oct 30 21:40:20.970953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:20.970889    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"]
Oct 30 21:40:21.471013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.458835    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:21.471013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.458922    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:21.505588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.505550    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"]
Oct 30 21:40:21.505822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.505764    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:21.598235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.598199    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:21.641080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.626485    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:21.641080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.626555    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:21.794773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.794731    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" event={"ID":"95892cb4-90b8-4996-8162-03da957b2b28","Type":"ContainerStarted","Data":"14df8933c8adc4e53305fd2f5b954db5a65d7aa4c828ed7d3449a02b6696bcbc"}
Oct 30 21:40:21.800684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:21.799758    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" event={"ID":"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6","Type":"ContainerStarted","Data":"512b322da04b7c4024d481bd565a4616575ebac0d042ca47c087b2fd9eb0c3db"}
Oct 30 21:40:22.479699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.472189    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:22.479699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.472259    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:22.600184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.599842    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:22.600184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.599899    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:22.838632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.835786    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" event={"ID":"95892cb4-90b8-4996-8162-03da957b2b28","Type":"ContainerStarted","Data":"f0876c04341fd31c4705f7274160c9cddef1dce70fd4d176a31261f9bfcb8d56"}
Oct 30 21:40:22.838632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.836987    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:22.873089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.873046    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" event={"ID":"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6","Type":"ContainerStarted","Data":"d12d0fed4e6dcf76b6aab04552896421e9ef12e83c94cae044d84e6d2e82aa34"}
Oct 30 21:40:22.885629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.877174    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:22.902215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.902111    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" podStartSLOduration=3.902078246 podStartE2EDuration="3.902078246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:22.901659376 +0000 UTC m=+1805.824117673" watchObservedRunningTime="2026-10-30 21:40:22.902078246 +0000 UTC m=+1805.824536546"
Oct 30 21:40:22.970632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:22.970470    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" podStartSLOduration=3.970444151 podStartE2EDuration="3.970444151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:22.950530438 +0000 UTC m=+1805.872988740" watchObservedRunningTime="2026-10-30 21:40:22.970444151 +0000 UTC m=+1805.892902461"
Oct 30 21:40:23.451523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:23.451482    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:23.451873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:23.451843    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:23.597577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:23.597522    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:23.598118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:23.597594    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:24.451629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.451040    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:24.451629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.451105    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:24.467912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.467855    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"]
Oct 30 21:40:24.473768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.473735    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"]
Oct 30 21:40:24.476644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.473876    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.479239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.479004    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869xb\" (UniqueName: \"kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.479239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.479056    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.479239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.479098    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.480240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.480218    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:24.482556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.482496    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"]
Oct 30 21:40:24.487872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.487840    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"]
Oct 30 21:40:24.490334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.488825    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:40:24.490334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.488896    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:40:24.490334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.489028    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:40:24.490334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.489105    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:40:24.500345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.500285    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-5494e8\""
Oct 30 21:40:24.500345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.500287    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-5494e8-dockercfg-2hdx4\""
Oct 30 21:40:24.816627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.799781    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869xb\" (UniqueName: \"kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.816627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.799871    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.816627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.799927    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.816627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.801583    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:24.994867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.991495    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:24.994867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.991648    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:24.996635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:24.996338    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-869xb\" (UniqueName: \"kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:25.082268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.082170    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-808058f8-fbbjn\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:25.084706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.084670    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"]
Oct 30 21:40:25.091554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.091522    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"]
Oct 30 21:40:25.091840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.091635    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.100820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.100788    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:25.102646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.102591    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzxz\" (UniqueName: \"kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz\") pod \"payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.102772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.102676    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.102772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.102733    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.102944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.102921    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgx6c\" (UniqueName: \"kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c\") pod \"payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.205364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.204473    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzxz\" (UniqueName: \"kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz\") pod \"payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.205364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.204530    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.205364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.204564    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.205364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.204643    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgx6c\" (UniqueName: \"kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c\") pod \"payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.206530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.206453    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.206530 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.206453    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.263572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.263447    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzxz\" (UniqueName: \"kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz\") pod \"payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.265303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.265230    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgx6c\" (UniqueName: \"kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c\") pod \"payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.434675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.419880    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:25.462444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.462403    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:25.462653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.462473    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:25.526155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.524902    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:25.578344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.578286    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"]
Oct 30 21:40:25.605705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.603867    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:25.605705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.603936    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:25.875638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:25.873142    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"]
Oct 30 21:40:25.879130 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:25.879079    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1276deaf_a385_4f4d_887c_8214e584cf68.slice/crio-485c788c97d91e5b9a8bff8c4a9ddaba7306b2cbceb386a9d8e280fc924d35be WatchSource:0}: Error finding container 485c788c97d91e5b9a8bff8c4a9ddaba7306b2cbceb386a9d8e280fc924d35be: Status 404 returned error can't find the container with id 485c788c97d91e5b9a8bff8c4a9ddaba7306b2cbceb386a9d8e280fc924d35be
Oct 30 21:40:26.048100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.037731    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:26.048100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.037801    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:26.072641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.069756    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"]
Oct 30 21:40:26.079880 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:26.079846    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb24b4a_0af8_491b_b6bb_7c5fe3353348.slice/crio-cab0dffc3316868599e82b92746eee5b4c56bd3e4b68df861f08c176bbf5ed10 WatchSource:0}: Error finding container cab0dffc3316868599e82b92746eee5b4c56bd3e4b68df861f08c176bbf5ed10: Status 404 returned error can't find the container with id cab0dffc3316868599e82b92746eee5b4c56bd3e4b68df861f08c176bbf5ed10
Oct 30 21:40:26.092958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.092925    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn" event={"ID":"fa7b2e82-ae7b-494d-8e54-fc4b9be48579","Type":"ContainerStarted","Data":"f49d3d27352bb88417d9eb176d51d6813744bd873b30170cb0aaed2c478b0810"}
Oct 30 21:40:26.094186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.094157    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" event={"ID":"0eb24b4a-0af8-491b-b6bb-7c5fe3353348","Type":"ContainerStarted","Data":"cab0dffc3316868599e82b92746eee5b4c56bd3e4b68df861f08c176bbf5ed10"}
Oct 30 21:40:26.095640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.095583    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" event={"ID":"1276deaf-a385-4f4d-887c-8214e584cf68","Type":"ContainerStarted","Data":"1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf"}
Oct 30 21:40:26.095766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.095665    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" event={"ID":"1276deaf-a385-4f4d-887c-8214e584cf68","Type":"ContainerStarted","Data":"485c788c97d91e5b9a8bff8c4a9ddaba7306b2cbceb386a9d8e280fc924d35be"}
Oct 30 21:40:26.096628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.096579    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:26.158677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.158593    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" podStartSLOduration=2.158571264 podStartE2EDuration="2.158571264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:26.158547863 +0000 UTC m=+1809.081006165" watchObservedRunningTime="2026-10-30 21:40:26.158571264 +0000 UTC m=+1809.081029557"
Oct 30 21:40:26.452486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.452025    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:26.452486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.452089    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:26.597764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.597723    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:26.597954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:26.597786    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:27.090895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.087703    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"]
Oct 30 21:40:27.094954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.094911    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"]
Oct 30 21:40:27.095892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.095868    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.176406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.176346    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" event={"ID":"0eb24b4a-0af8-491b-b6bb-7c5fe3353348","Type":"ContainerStarted","Data":"074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61"}
Oct 30 21:40:27.176722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.176701    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:27.180874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.180831    6695 generic.go:356] "Generic (PLEG): container finished" podID="fa7b2e82-ae7b-494d-8e54-fc4b9be48579" containerID="b3a79bd164f90ffce0bb96073c8c06172e63c333526a163023e7a563644dad55" exitCode=0
Oct 30 21:40:27.181220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.181156    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn" event={"ID":"fa7b2e82-ae7b-494d-8e54-fc4b9be48579","Type":"ContainerDied","Data":"b3a79bd164f90ffce0bb96073c8c06172e63c333526a163023e7a563644dad55"}
Oct 30 21:40:27.198943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.198893    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj64z\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199117    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199184    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199223    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199452    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199551    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.199675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199650    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.200004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.199874    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.295025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.294916    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" podStartSLOduration=3.294897767 podStartE2EDuration="3.294897767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:27.250873195 +0000 UTC m=+1810.173331493" watchObservedRunningTime="2026-10-30 21:40:27.294897767 +0000 UTC m=+1810.217356065"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.307926    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.316748    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.316872    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj64z\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.316966    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.317056    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.317101    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.317147    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.317227    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.317305    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.318453    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.319161    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.328477    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.328799    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.336776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.329504    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.400343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.399838    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.400343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.400263    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj64z\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.400343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.400333    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.403521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.403445    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.457631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.451150    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:40:27.457631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.451233    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Oct 30 21:40:27.482995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.482892    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:27.622013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.610873    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:27.622013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.610939    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:27.855850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.855698    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:27.855850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.855805    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:27.855850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.855851    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:27.856322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:27.856288    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"]
Oct 30 21:40:28.199051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.198484    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" event={"ID":"0e8aa9e3-4606-4ec8-acf2-7e044a590f72","Type":"ContainerStarted","Data":"0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8"}
Oct 30 21:40:28.199051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.198553    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" event={"ID":"0e8aa9e3-4606-4ec8-acf2-7e044a590f72","Type":"ContainerStarted","Data":"820c57d5f82dd2e29db386f43afc69c66752074d4fed98fa3959c45e26e9955a"}
Oct 30 21:40:28.250902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.250833    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podStartSLOduration=1.250808216 podStartE2EDuration="1.250808216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:28.245920245 +0000 UTC m=+1811.168378543" watchObservedRunningTime="2026-10-30 21:40:28.250808216 +0000 UTC m=+1811.173266516"
Oct 30 21:40:28.516624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.511272    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:28.516624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.515326    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:28.532628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.522003    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:28.532628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.522096    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:28.603641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.594192    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca\") pod \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") "
Oct 30 21:40:28.603641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.594367    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb\" (UniqueName: \"kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb\") pod \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") "
Oct 30 21:40:28.603641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.594461    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca\") pod \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\" (UID: \"fa7b2e82-ae7b-494d-8e54-fc4b9be48579\") "
Oct 30 21:40:28.604994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.604952    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca" pod "fa7b2e82-ae7b-494d-8e54-fc4b9be48579" (UID: "fa7b2e82-ae7b-494d-8e54-fc4b9be48579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:28.638716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.614720    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:40:28.638716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.614796    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Oct 30 21:40:28.638716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.621793    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb" pod "fa7b2e82-ae7b-494d-8e54-fc4b9be48579" (UID: "fa7b2e82-ae7b-494d-8e54-fc4b9be48579"). InnerVolumeSpecName "kube-api-access-869xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:28.683886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.661424    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca" pod "fa7b2e82-ae7b-494d-8e54-fc4b9be48579" (UID: "fa7b2e82-ae7b-494d-8e54-fc4b9be48579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:28.716641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.706856    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:28.716641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.706892    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869xb\" (UniqueName: \"kubernetes.io/projected/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-kube-api-access-869xb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:28.716641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:28.706907    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa7b2e82-ae7b-494d-8e54-fc4b9be48579-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:40:29.360637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.359780    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"
Oct 30 21:40:29.365717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.365672    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn" event={"ID":"fa7b2e82-ae7b-494d-8e54-fc4b9be48579","Type":"ContainerDied","Data":"f49d3d27352bb88417d9eb176d51d6813744bd873b30170cb0aaed2c478b0810"}
Oct 30 21:40:29.366027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.366005    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f49d3d27352bb88417d9eb176d51d6813744bd873b30170cb0aaed2c478b0810"
Oct 30 21:40:29.495922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.492006    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:29.495922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.492071    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:29.499127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.499079    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:29.499454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:29.499400    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:30.017629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.007083    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:30.103629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.103020    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:30.336629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.335697    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:30.340624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.337432    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:30.347627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.345891    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:40:30.493638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.491235    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:30.493638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.491329    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:30.628692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.628507    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:30.628692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.628587    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:30.859642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:30.836109    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.116793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.114676    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.116793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.115206    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:40:31.116793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.116430    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:40:31.320034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.319751    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.495495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.495387    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:31.520636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.513856    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:31.662678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.648114    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:31.662678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.648188    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.684054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.684013    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:31.687087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.684333    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.758145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.687466    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:31.758145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.722313    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:40:31.910634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.909738    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 21:40:31.973737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:31.940323    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 21:40:31.993712 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:40:31.989789    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:40:32.389960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.389911    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:40:32.390887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.390852    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:40:32.506634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.496754    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:32.506634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.496823    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:32.614640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.612969    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:32.839791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.838188    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 21:40:32.910912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.910848    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:32.920637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:32.915501    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 21:40:33.483873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:33.483825    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:33.484079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:33.483937    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:33.889503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:33.889472    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 21:40:34.484120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:34.484075    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:34.484325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:34.484161    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:35.086382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:35.086348    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 21:40:35.483796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:35.483760    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:35.484052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:35.483846    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:36.232630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.221509    6695 generic.go:356] "Generic (PLEG): container finished" podID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerID="2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436" exitCode=0
Oct 30 21:40:36.232630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.221893    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" event={"ID":"2c7f4209-3fba-41a2-b098-b81984be14cf","Type":"ContainerDied","Data":"2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"}
Oct 30 21:40:36.232630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.222455    6695 scope.go:122] "RemoveContainer" containerID="2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"
Oct 30 21:40:36.257016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.256953    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:36.258046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.258016    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:36.259743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.258248    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:36.483983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.483884    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Oct 30 21:40:36.483983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:36.483954    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Oct 30 21:40:37.259626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:37.258663    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" event={"ID":"2c7f4209-3fba-41a2-b098-b81984be14cf","Type":"ContainerStarted","Data":"a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0"}
Oct 30 21:40:38.189424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.189386    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:40:38.190264 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.190232    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:40:38.190408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.190298    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:40:38.202483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.202442    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 21:40:38.202733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.202538    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 21:40:38.577648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.573830    6695 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:38.577648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:38.573937    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:39.189017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:39.188951    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:40:39.189235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:39.189041    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:40:39.200043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:39.200004    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:39.200564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:39.200535    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:39.201506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:39.201486    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:40:39.936631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:39.929719    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:40:40.190073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:40.189972    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:40:40.190073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:40.190038    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:40:41.189454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.189414    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:40:41.190026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.189470    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:40:41.986874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.986837    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"]
Oct 30 21:40:41.987584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.987564    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fa7b2e82-ae7b-494d-8e54-fc4b9be48579" containerName="revoke-keys"
Oct 30 21:40:41.987584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.987584    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7b2e82-ae7b-494d-8e54-fc4b9be48579" containerName="revoke-keys"
Oct 30 21:40:41.987852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.987836    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7b2e82-ae7b-494d-8e54-fc4b9be48579" containerName="revoke-keys"
Oct 30 21:40:41.991640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:41.991585    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.006128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.006100    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e06b90\""
Oct 30 21:40:42.025860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.025825    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e06b90-dockercfg-9lpzx\""
Oct 30 21:40:42.061630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.051038    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"]
Oct 30 21:40:42.061630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.052328    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96ft\" (UniqueName: \"kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft\") pod \"payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.061630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.052385    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.165869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.165819    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96ft\" (UniqueName: \"kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft\") pod \"payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.166192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.166165    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.176805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.176771    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.219984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.219946    6695 patch_prober.go:29] interesting pod/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body=
Oct 30 21:40:42.220659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.220616    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused"
Oct 30 21:40:42.298846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.298806    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96ft\" (UniqueName: \"kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft\") pod \"payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.327643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.327300    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:42.834863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:42.834829    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"]
Oct 30 21:40:42.835643 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:42.835589    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640adee0_832b_4e53_9eb9_22fe76a9f561.slice/crio-e846d78eca75408991cdf683b8218b3f8fc8dead920ce9018b44eef5726a58a2 WatchSource:0}: Error finding container e846d78eca75408991cdf683b8218b3f8fc8dead920ce9018b44eef5726a58a2: Status 404 returned error can't find the container with id e846d78eca75408991cdf683b8218b3f8fc8dead920ce9018b44eef5726a58a2
Oct 30 21:40:43.193927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:43.193853    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:40:43.195266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:43.195242    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:40:43.511180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:43.511080    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" event={"ID":"640adee0-832b-4e53-9eb9-22fe76a9f561","Type":"ContainerStarted","Data":"601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db"}
Oct 30 21:40:43.511180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:43.511147    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" event={"ID":"640adee0-832b-4e53-9eb9-22fe76a9f561","Type":"ContainerStarted","Data":"e846d78eca75408991cdf683b8218b3f8fc8dead920ce9018b44eef5726a58a2"}
Oct 30 21:40:43.555013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:43.554943    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" podStartSLOduration=2.554907738 podStartE2EDuration="2.554907738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:43.553441009 +0000 UTC m=+1826.475899310" watchObservedRunningTime="2026-10-30 21:40:43.554907738 +0000 UTC m=+1826.477366035"
Oct 30 21:40:44.519189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:44.519154    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:55.537780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:55.537742    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 21:40:58.060102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.055848    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"]
Oct 30 21:40:58.094240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.093955    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"]
Oct 30 21:40:58.094240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.094069    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.175636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.167004    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-bfabe1-openshift-default-dockercfg-rjxpg\""
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.227636    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.227765    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.227804    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.227850    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.227932    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7kq\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.228015    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.228086    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.228133    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.245630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.228214    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.330749    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.330848    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.330885    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.330930    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.331041    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7kq\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.331134    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.331208    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.331264    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.331690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.331349    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.332463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.332292    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.335629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.332543    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.335629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.332952    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.367371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.337530    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.367371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.338534    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.367371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.357932    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.385646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.379712    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.385646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.380418    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.389897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.388595    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7kq\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq\") pod \"e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") " pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.427628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.426898    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:40:58.910475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.910447    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"]
Oct 30 21:40:58.947631 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:40:58.946735    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60f2c1e_2764_4e72_9ba2_24a821a9c2cb.slice/crio-e7d28b4a440ca520d52318a079716203eed05430831b1a78f3b5a64d7eb11188 WatchSource:0}: Error finding container e7d28b4a440ca520d52318a079716203eed05430831b1a78f3b5a64d7eb11188: Status 404 returned error can't find the container with id e7d28b4a440ca520d52318a079716203eed05430831b1a78f3b5a64d7eb11188
Oct 30 21:40:58.966062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.955980    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:58.966062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.956081    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:58.966062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:58.956126    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:59.891257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:59.891221    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" event={"ID":"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb","Type":"ContainerStarted","Data":"4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"}
Oct 30 21:40:59.891719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:59.891286    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" event={"ID":"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb","Type":"ContainerStarted","Data":"e7d28b4a440ca520d52318a079716203eed05430831b1a78f3b5a64d7eb11188"}
Oct 30 21:40:59.967482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:40:59.967409    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podStartSLOduration=1.9673860429999999 podStartE2EDuration="1.967386043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:59.959189396 +0000 UTC m=+1842.881647694" watchObservedRunningTime="2026-10-30 21:40:59.967386043 +0000 UTC m=+1842.889844341"
Oct 30 21:41:00.433633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:00.431583    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:41:00.444669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:00.444636    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:41:00.988161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:00.988121    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:41:00.993457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:00.993425    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:41:01.142897 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:41:01.142502    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:41:04.287986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.287944    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"]
Oct 30 21:41:04.294495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.294462    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.325995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.322227    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-bfabe1-dockercfg-v6794\""
Oct 30 21:41:04.327305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.327265    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"]
Oct 30 21:41:04.462388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.462352    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-bfabe1\""
Oct 30 21:41:04.533712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.482935    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.534281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.534249    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vzq\" (UniqueName: \"kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq\") pod \"payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.537259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.537225    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"]
Oct 30 21:41:04.546115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.546030    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.559937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.559900    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"]
Oct 30 21:41:04.643887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.643843    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.644577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.643953    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.644577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.644156    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vzq\" (UniqueName: \"kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq\") pod \"payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.644577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.644190    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxghh\" (UniqueName: \"kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh\") pod \"payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.649954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.646217    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.666266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.666227    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:41:04.666493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.666281    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:04.667781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.667744    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vzq\" (UniqueName: \"kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq\") pod \"payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.761329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.761278    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxghh\" (UniqueName: \"kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh\") pod \"payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.761579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.761471    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.762374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.762346    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.780994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.780951    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxghh\" (UniqueName: \"kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh\") pod \"payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:04.914413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.914374    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:04.969128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:04.969086    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:05.445584 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:41:05.445550    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4933a4_2dc2_41f1_978c_56b2b30694b2.slice/crio-ce008a1ef7fff67d71ab23f944772372641a820d4a5bfb84d96cbb20ebf520c1 WatchSource:0}: Error finding container ce008a1ef7fff67d71ab23f944772372641a820d4a5bfb84d96cbb20ebf520c1: Status 404 returned error can't find the container with id ce008a1ef7fff67d71ab23f944772372641a820d4a5bfb84d96cbb20ebf520c1
Oct 30 21:41:05.445956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:05.445712    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"]
Oct 30 21:41:05.545988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:05.545944    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"]
Oct 30 21:41:05.549998 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:41:05.549959    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25652f6_e0f8_4960_8d1b_59b5686054c6.slice/crio-129addf1a7b1ac3beb2d42dc279d31a8bdc05a8b7d1f47766f4f2bc8f1c2e484 WatchSource:0}: Error finding container 129addf1a7b1ac3beb2d42dc279d31a8bdc05a8b7d1f47766f4f2bc8f1c2e484: Status 404 returned error can't find the container with id 129addf1a7b1ac3beb2d42dc279d31a8bdc05a8b7d1f47766f4f2bc8f1c2e484
Oct 30 21:41:06.280449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.280413    6695 generic.go:356] "Generic (PLEG): container finished" podID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerID="10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0" exitCode=0
Oct 30 21:41:06.280671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.280487    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" event={"ID":"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73","Type":"ContainerDied","Data":"10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"}
Oct 30 21:41:06.281086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.281054    6695 scope.go:122] "RemoveContainer" containerID="10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"
Oct 30 21:41:06.282913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.282884    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" event={"ID":"a25652f6-e0f8-4960-8d1b-59b5686054c6","Type":"ContainerStarted","Data":"37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119"}
Oct 30 21:41:06.283043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.282945    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" event={"ID":"a25652f6-e0f8-4960-8d1b-59b5686054c6","Type":"ContainerStarted","Data":"129addf1a7b1ac3beb2d42dc279d31a8bdc05a8b7d1f47766f4f2bc8f1c2e484"}
Oct 30 21:41:06.283043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.282994    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:06.283164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.283031    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:06.283164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.283120    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:06.283282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.283168    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:06.285724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.285691    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" event={"ID":"8d4933a4-2dc2-41f1-978c-56b2b30694b2","Type":"ContainerStarted","Data":"657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960"}
Oct 30 21:41:06.285852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.285754    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" event={"ID":"8d4933a4-2dc2-41f1-978c-56b2b30694b2","Type":"ContainerStarted","Data":"ce008a1ef7fff67d71ab23f944772372641a820d4a5bfb84d96cbb20ebf520c1"}
Oct 30 21:41:06.285852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.285840    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:06.368808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.368734    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" podStartSLOduration=2.368714061 podStartE2EDuration="2.368714061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:06.368278511 +0000 UTC m=+1849.290736827" watchObservedRunningTime="2026-10-30 21:41:06.368714061 +0000 UTC m=+1849.291172376"
Oct 30 21:41:06.414241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:06.413362    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" podStartSLOduration=2.413341923 podStartE2EDuration="2.413341923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:06.405000419 +0000 UTC m=+1849.327458717" watchObservedRunningTime="2026-10-30 21:41:06.413341923 +0000 UTC m=+1849.335800221"
Oct 30 21:41:07.297303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:07.297254    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" event={"ID":"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73","Type":"ContainerStarted","Data":"40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6"}
Oct 30 21:41:07.717684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:07.714700    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:41:07.717684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:07.714879    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:41:07.717684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:07.714923    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 21:41:08.096553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.096514    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"]
Oct 30 21:41:08.100461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.100428    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.158961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.158920    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"]
Oct 30 21:41:08.161091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161134    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161242    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161279    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv29k\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161312    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161379    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161403    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161456    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.161560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.161503    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.262806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.262768    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.262913    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.262979    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263031    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263071    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv29k\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263113    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263267    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263321    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263424    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263421    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263715    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263777    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.263914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.263895    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.264297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.264272    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.265657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.265576    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.266087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.266067    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.282964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.282926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.283234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.283193    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv29k\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.453386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.453278    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:08.734698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.734042    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:41:08.734698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.734108    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 21:41:08.866465 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.866349    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"]
Oct 30 21:41:08.877589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.876670    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:08.877589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.876772    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:08.877589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:08.876823    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:09.371165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.371046    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" event={"ID":"57bbe453-77e5-454a-88c1-073ee6b6e94c","Type":"ContainerStarted","Data":"b523830c067d1a3bc89aecd02e300f6d4862f97393c64b97f10b224dd29ac7ea"}
Oct 30 21:41:09.374591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.374551    6695 generic.go:356] "Generic (PLEG): container finished" podID="77bfd09c-2941-477e-8e75-f06def9a9920" containerID="d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a" exitCode=0
Oct 30 21:41:09.374789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.374774    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" event={"ID":"77bfd09c-2941-477e-8e75-f06def9a9920","Type":"ContainerDied","Data":"d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"}
Oct 30 21:41:09.375317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.375297    6695 scope.go:122] "RemoveContainer" containerID="d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"
Oct 30 21:41:09.377851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.377731    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:09.377851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.377828    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:09.378023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.377871    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:09.724701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.715091    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:41:09.724701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:09.715170    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 21:41:10.047640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.039884    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:41:10.385671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.385534    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" event={"ID":"77bfd09c-2941-477e-8e75-f06def9a9920","Type":"ContainerStarted","Data":"6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e"}
Oct 30 21:41:10.387535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.387502    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" event={"ID":"57bbe453-77e5-454a-88c1-073ee6b6e94c","Type":"ContainerStarted","Data":"5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787"}
Oct 30 21:41:10.453492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.453445    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:10.455939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.455903    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:10.456119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.455973    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:10.481969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.481891    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podStartSLOduration=2.481869467 podStartE2EDuration="2.481869467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:10.48046529 +0000 UTC m=+1853.402923589" watchObservedRunningTime="2026-10-30 21:41:10.481869467 +0000 UTC m=+1853.404327765"
Oct 30 21:41:10.730688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.712678    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:41:10.730688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:10.712742    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 21:41:11.175657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.167029    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:41:11.175657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.174757    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:11.175657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.174838    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:11.454688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.454024    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:11.454688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.454092    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:11.718693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.715171    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:41:11.718693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:11.715251    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 21:41:12.172628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:12.172055    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:12.172628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:12.172121    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:12.468695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:12.468029    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:12.468695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:12.468103    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:12.731710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:12.726103    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:41:12.731710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:12.726186    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Oct 30 21:41:13.172403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.172352    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:13.172708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.172425    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:13.454688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.454201    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:13.454688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.454286    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:13.750692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.750019    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:41:13.754632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.751880    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:41:13.767683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.766302    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 21:41:14.001630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:13.997784    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"]
Oct 30 21:41:14.004873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.004621    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"]
Oct 30 21:41:14.004873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.004761    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.056779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.038535    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-ae6492-openshift-default-dockercfg-xlpqc\""
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.096767    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.096881    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.096980    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7pk\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.097013    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.097052    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.097146    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.097242    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.097372    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.097668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.097398    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.185517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.185467    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:14.186069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.185553    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:14.200099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200044    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7pk\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200114    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200154    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200239    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200425    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200459    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200582    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.200709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.200692    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.201327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.201300    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.203032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.202994    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.203304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.203277    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.203544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.203512    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.204475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.204441    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.206682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.206598    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.207262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.207223    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.270818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.270716    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7pk\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.276553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.273759    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.457624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.453895    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:14.457624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.456828    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:14.457624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.456903    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:14.830630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.727690    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"]
Oct 30 21:41:14.872647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.871969    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:14.936900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.892997    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f49db2-dockercfg-rrpql\""
Oct 30 21:41:14.936900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.909550    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f49db2\""
Oct 30 21:41:14.987875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.953544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:14.987875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:14.953878    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqh98\" (UniqueName: \"kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98\") pod \"payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:15.004433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.003515    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"]
Oct 30 21:41:15.068885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.068831    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqh98\" (UniqueName: \"kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98\") pod \"payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:15.069261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.069233    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:15.078948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.070283    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:15.098985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.098948    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqh98\" (UniqueName: \"kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98\") pod \"payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:15.169659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.169620    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:15.169983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.169936    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:15.291932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.291892    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:15.397631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.397585    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"]
Oct 30 21:41:15.407409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.406247    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:15.407409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.406344    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:15.407409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.406398    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:15.470860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.470812    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:15.471054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.470904    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:15.775121 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:41:15.775084    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e70a689_84d1_4aee_84c9_fe40c8b801d4.slice/crio-e802d86d185aee4512a3805afe33b581395f32633002072eb56905fce03510d3 WatchSource:0}: Error finding container e802d86d185aee4512a3805afe33b581395f32633002072eb56905fce03510d3: Status 404 returned error can't find the container with id e802d86d185aee4512a3805afe33b581395f32633002072eb56905fce03510d3
Oct 30 21:41:15.776453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.776405    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"]
Oct 30 21:41:15.787154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.787061    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" event={"ID":"1cb42d7c-2086-4352-a6b3-3e8f3c09da77","Type":"ContainerStarted","Data":"e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"}
Oct 30 21:41:15.787154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:15.787111    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" event={"ID":"1cb42d7c-2086-4352-a6b3-3e8f3c09da77","Type":"ContainerStarted","Data":"bd2befcad7ae98ec263ab6b37c1a23d4aa3109c075358ddcc44e79d459ed7402"}
Oct 30 21:41:16.169595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.169044    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:16.169595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.169113    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:16.454740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.454450    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Oct 30 21:41:16.454740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.454515    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Oct 30 21:41:16.474394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.458580    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:16.480550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.480514    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:16.561931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.561868    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podStartSLOduration=3.561844926 podStartE2EDuration="3.561844926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:15.865012569 +0000 UTC m=+1858.787470869" watchObservedRunningTime="2026-10-30 21:41:16.561844926 +0000 UTC m=+1859.484303227"
Oct 30 21:41:16.855616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.855562    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" event={"ID":"2e70a689-84d1-4aee-84c9-fe40c8b801d4","Type":"ContainerStarted","Data":"bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a"}
Oct 30 21:41:16.855931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.855904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" event={"ID":"2e70a689-84d1-4aee-84c9-fe40c8b801d4","Type":"ContainerStarted","Data":"e802d86d185aee4512a3805afe33b581395f32633002072eb56905fce03510d3"}
Oct 30 21:41:16.856067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.856054    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:16.856204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.856193    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:16.870493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.867944    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:41:16.903796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:16.903727    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" podStartSLOduration=2.903701547 podStartE2EDuration="2.903701547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:16.893134428 +0000 UTC m=+1859.815592728" watchObservedRunningTime="2026-10-30 21:41:16.903701547 +0000 UTC m=+1859.826159847"
Oct 30 21:41:17.197858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.197761    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:41:17.198124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.198092    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Oct 30 21:41:17.391421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.391374    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 21:41:17.391679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.391465    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 21:41:17.664261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.664218    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:17.666002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.665974    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:17.667447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:17.667418    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:41:18.168334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:18.168289    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:41:18.168826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:18.168799    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:41:18.169672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:18.169650    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 21:41:20.407860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.407825    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"]
Oct 30 21:41:20.436284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.436249    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.441373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.441341    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"]
Oct 30 21:41:20.467176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.467134    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-ae6492\""
Oct 30 21:41:20.467375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.467258    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-ae6492-dockercfg-gvgmh\""
Oct 30 21:41:20.468933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.468881    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.469292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.469267    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85\") pod \"payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.593635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.591448    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85\") pod \"payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.656694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.643400    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.668687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.668389    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.701627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.680426    6695 scope.go:122] "RemoveContainer" containerID="4facad9d92336adf9ecf108c4dbbd6a200bd48bf85940c0e0417ef43af62727c"
Oct 30 21:41:20.851493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.851440    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85\") pod \"payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:20.877438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.877411    6695 scope.go:122] "RemoveContainer" containerID="61ef3e18d4a757ab9bcfca29d0c43ff7aa768292bb561b1d24592492b463673d"
Oct 30 21:41:20.892579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:20.892514    6695 scope.go:122] "RemoveContainer" containerID="3b37fcd799fdd2eb4e5cd1a0d92cc20e8df962c055a08cfa767da818a0ef30d0"
Oct 30 21:41:21.055495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:21.055462    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:21.406124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:21.406084    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"]
Oct 30 21:41:21.407980 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:41:21.407954    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67835098_f7b5_45f1_bdb6_339d0ba1578e.slice/crio-8219b84c52abd1ecbc5195cab0ab34b7a685de022d767a01ba555981faa40e4d WatchSource:0}: Error finding container 8219b84c52abd1ecbc5195cab0ab34b7a685de022d767a01ba555981faa40e4d: Status 404 returned error can't find the container with id 8219b84c52abd1ecbc5195cab0ab34b7a685de022d767a01ba555981faa40e4d
Oct 30 21:41:21.973631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:21.973418    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl" event={"ID":"67835098-f7b5-45f1-bdb6-339d0ba1578e","Type":"ContainerStarted","Data":"8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30"}
Oct 30 21:41:21.973631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:21.973491    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl" event={"ID":"67835098-f7b5-45f1-bdb6-339d0ba1578e","Type":"ContainerStarted","Data":"8219b84c52abd1ecbc5195cab0ab34b7a685de022d767a01ba555981faa40e4d"}
Oct 30 21:41:21.978953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:21.978919    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:22.040714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:22.038419    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl" podStartSLOduration=2.038398351 podStartE2EDuration="2.038398351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:22.038042161 +0000 UTC m=+1864.960500460" watchObservedRunningTime="2026-10-30 21:41:22.038398351 +0000 UTC m=+1864.960856651"
Oct 30 21:41:28.886723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:28.886689    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 21:41:29.172378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:29.172266    6695 generic.go:356] "Generic (PLEG): container finished" podID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerID="75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35" exitCode=0
Oct 30 21:41:29.172809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:29.172778    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" event={"ID":"e4c4c2b4-4a4b-41d6-aa81-66f0da709615","Type":"ContainerDied","Data":"75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"}
Oct 30 21:41:29.173466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:29.173437    6695 scope.go:122] "RemoveContainer" containerID="75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"
Oct 30 21:41:29.179935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:29.177962    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:29.179935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:29.178072    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:29.179935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:29.178124    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:30.182811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:30.182766    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" event={"ID":"e4c4c2b4-4a4b-41d6-aa81-66f0da709615","Type":"ContainerStarted","Data":"129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912"}
Oct 30 21:41:30.636955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:30.636909    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:41:30.637199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:30.637172    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 21:41:30.637287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:30.637229    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 21:41:31.076218 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:41:31.076184    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:41:31.638101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:31.638054    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 21:41:31.638686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:31.638127    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 21:41:32.637630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:32.637311    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 21:41:32.637630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:32.637374    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 21:41:33.639372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:33.639327    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 21:41:33.639956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:33.639407    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 21:41:34.042726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:34.042674    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 21:41:34.637929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:34.637883    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 21:41:34.638163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:34.637959    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 21:41:35.637463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:35.637423    6695 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Oct 30 21:41:35.638202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:35.638166    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Oct 30 21:41:35.714210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:35.714134    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:41:36.643389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:36.643350    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:41:36.644794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:36.644768    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 21:41:37.262331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.262291    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"]
Oct 30 21:41:37.267444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.267411    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.298273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.298224    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-64d142\""
Oct 30 21:41:37.306410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.306373    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"]
Oct 30 21:41:37.307450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.307421    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-64d142-dockercfg-8w9jk\""
Oct 30 21:41:37.309336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.309301    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.309846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.309825    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwghb\" (UniqueName: \"kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb\") pod \"payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.357213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.357167    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"]
Oct 30 21:41:37.363825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.363790    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.395250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.395185    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"]
Oct 30 21:41:37.411848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.411765    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvr6r\" (UniqueName: \"kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r\") pod \"payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.412075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.411929    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.412075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.412042    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwghb\" (UniqueName: \"kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb\") pod \"payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.412203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.412101    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.413316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.413287    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.431763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.431729    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwghb\" (UniqueName: \"kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb\") pod \"payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.532633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.523934    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvr6r\" (UniqueName: \"kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r\") pod \"payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.532633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.524012    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.532633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.525455    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.588237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.586925    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:37.591624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.589301    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvr6r\" (UniqueName: \"kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r\") pod \"payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:37.696633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:37.692105    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:38.126359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.126322    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"]
Oct 30 21:41:38.128710 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:41:38.128677    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8bded2_3c51_4e1d_a1b9_729171e6596c.slice/crio-cd141439d5e8b09adccd216b1187ba329e1ca61edff892e0cede1e74ff727271 WatchSource:0}: Error finding container cd141439d5e8b09adccd216b1187ba329e1ca61edff892e0cede1e74ff727271: Status 404 returned error can't find the container with id cd141439d5e8b09adccd216b1187ba329e1ca61edff892e0cede1e74ff727271
Oct 30 21:41:38.140634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.140579    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"]
Oct 30 21:41:38.146456 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:41:38.146424    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52929c6_7e60_47f8_9861_1f277637b1c8.slice/crio-7ddcd3ea916ed7e52bdd7f6d559b9e8ec79e8a84871ff5a112337b0345422291 WatchSource:0}: Error finding container 7ddcd3ea916ed7e52bdd7f6d559b9e8ec79e8a84871ff5a112337b0345422291: Status 404 returned error can't find the container with id 7ddcd3ea916ed7e52bdd7f6d559b9e8ec79e8a84871ff5a112337b0345422291
Oct 30 21:41:38.294296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.294259    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" event={"ID":"c52929c6-7e60-47f8-9861-1f277637b1c8","Type":"ContainerStarted","Data":"84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8"}
Oct 30 21:41:38.294502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.294317    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" event={"ID":"c52929c6-7e60-47f8-9861-1f277637b1c8","Type":"ContainerStarted","Data":"7ddcd3ea916ed7e52bdd7f6d559b9e8ec79e8a84871ff5a112337b0345422291"}
Oct 30 21:41:38.294502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.294345    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:38.296466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.296439    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" event={"ID":"0a8bded2-3c51-4e1d-a1b9-729171e6596c","Type":"ContainerStarted","Data":"0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2"}
Oct 30 21:41:38.296739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.296490    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:38.296739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.296512    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" event={"ID":"0a8bded2-3c51-4e1d-a1b9-729171e6596c","Type":"ContainerStarted","Data":"cd141439d5e8b09adccd216b1187ba329e1ca61edff892e0cede1e74ff727271"}
Oct 30 21:41:38.331911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.331839    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" podStartSLOduration=1.331818901 podStartE2EDuration="1.331818901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:38.329886972 +0000 UTC m=+1881.252345281" watchObservedRunningTime="2026-10-30 21:41:38.331818901 +0000 UTC m=+1881.254277198"
Oct 30 21:41:38.367534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:38.367469    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" podStartSLOduration=1.367451677 podStartE2EDuration="1.367451677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:38.365847963 +0000 UTC m=+1881.288306261" watchObservedRunningTime="2026-10-30 21:41:38.367451677 +0000 UTC m=+1881.289910065"
Oct 30 21:41:49.378229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:49.374828    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 21:41:49.378229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:49.376683    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 21:41:52.262902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.262857    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"]
Oct 30 21:41:52.269164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.269132    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.284620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.284497    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"]
Oct 30 21:41:52.289934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.289898    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:41:52.289934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.289925    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:41:52.290161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.289916    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:41:52.293956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.293928    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:41:52.397821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.397776    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.398016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.397948    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.398086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.398067    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdd9d\" (UniqueName: \"kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.500285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.500243    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.500515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.500340    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.500515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.500405    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdd9d\" (UniqueName: \"kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.501115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.501084    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.502756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.502731    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.542626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.539451    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdd9d\" (UniqueName: \"kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d\") pod \"maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.587760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.587727    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:52.956686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:52.954315    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"]
Oct 30 21:41:53.601436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:53.601384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh" event={"ID":"4f86b536-58f7-4170-a836-12ebecc66f4f","Type":"ContainerStarted","Data":"737e365af361874fb7b1d1f796a8d5cec1a116448a97ad773413f61de26dee8d"}
Oct 30 21:41:53.601906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:53.601456    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh" event={"ID":"4f86b536-58f7-4170-a836-12ebecc66f4f","Type":"ContainerStarted","Data":"b329db170e668c3d90b1f38db9997496499a0d61f9592ffcb130242b67e17fc2"}
Oct 30 21:41:53.647450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:53.647374    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh" podStartSLOduration=1.647354582 podStartE2EDuration="1.647354582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:53.644437699 +0000 UTC m=+1896.566895997" watchObservedRunningTime="2026-10-30 21:41:53.647354582 +0000 UTC m=+1896.569812882"
Oct 30 21:41:54.636667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:54.630766    6695 generic.go:356] "Generic (PLEG): container finished" podID="4f86b536-58f7-4170-a836-12ebecc66f4f" containerID="737e365af361874fb7b1d1f796a8d5cec1a116448a97ad773413f61de26dee8d" exitCode=0
Oct 30 21:41:54.636667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:54.630906    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh" event={"ID":"4f86b536-58f7-4170-a836-12ebecc66f4f","Type":"ContainerDied","Data":"737e365af361874fb7b1d1f796a8d5cec1a116448a97ad773413f61de26dee8d"}
Oct 30 21:41:55.859793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.859763    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:55.888710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.888669    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca\") pod \"4f86b536-58f7-4170-a836-12ebecc66f4f\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") "
Oct 30 21:41:55.888907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.888720    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d\" (UniqueName: \"kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d\") pod \"4f86b536-58f7-4170-a836-12ebecc66f4f\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") "
Oct 30 21:41:55.888907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.888783    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca\") pod \"4f86b536-58f7-4170-a836-12ebecc66f4f\" (UID: \"4f86b536-58f7-4170-a836-12ebecc66f4f\") "
Oct 30 21:41:55.889232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.889204    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca" pod "4f86b536-58f7-4170-a836-12ebecc66f4f" (UID: "4f86b536-58f7-4170-a836-12ebecc66f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:55.889518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.889501    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:41:55.890976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.890949    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca" pod "4f86b536-58f7-4170-a836-12ebecc66f4f" (UID: "4f86b536-58f7-4170-a836-12ebecc66f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:41:55.891091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.891067    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d" pod "4f86b536-58f7-4170-a836-12ebecc66f4f" (UID: "4f86b536-58f7-4170-a836-12ebecc66f4f"). InnerVolumeSpecName "kube-api-access-xdd9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:55.991291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.991249    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f86b536-58f7-4170-a836-12ebecc66f4f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:41:55.991291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:55.991284    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdd9d\" (UniqueName: \"kubernetes.io/projected/4f86b536-58f7-4170-a836-12ebecc66f4f-kube-api-access-xdd9d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:41:56.654540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:56.654505    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh" event={"ID":"4f86b536-58f7-4170-a836-12ebecc66f4f","Type":"ContainerDied","Data":"b329db170e668c3d90b1f38db9997496499a0d61f9592ffcb130242b67e17fc2"}
Oct 30 21:41:56.654540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:56.654530    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"
Oct 30 21:41:56.654811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:41:56.654582    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b329db170e668c3d90b1f38db9997496499a0d61f9592ffcb130242b67e17fc2"
Oct 30 21:42:00.826893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.826807    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"]
Oct 30 21:42:00.827724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.827705    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4f86b536-58f7-4170-a836-12ebecc66f4f" containerName="revoke-keys"
Oct 30 21:42:00.827781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.827725    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f86b536-58f7-4170-a836-12ebecc66f4f" containerName="revoke-keys"
Oct 30 21:42:00.827878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.827867    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f86b536-58f7-4170-a836-12ebecc66f4f" containerName="revoke-keys"
Oct 30 21:42:00.833073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.833042    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.850786    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.850824    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.850970    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.851121    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqzt\" (UniqueName: \"kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.851219    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.851298    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.851668    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"]
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.857039    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-09eabe\"/\"kube-root-ca.crt\""
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.857137    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-09eabe\"/\"openshift-service-ca.crt\""
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.857262    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-09eabe\"/\"auth-test-model-09eabe-kserve-self-signed-certs\""
Oct 30 21:42:00.861859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.861767    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-09eabe\"/\"default-dockercfg-kn5dc\""
Oct 30 21:42:00.954619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.954568    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.954909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.954627    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.954909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.954712    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.955087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.955061    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqzt\" (UniqueName: \"kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.955152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.955131    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.955212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.955149    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.955271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.955211    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.955674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.955581    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.956105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.956077    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.958862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.958840    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.959229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.959207    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.976829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.976796    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqzt\" (UniqueName: \"kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt\") pod \"auth-test-model-09eabe-kserve-55bf7d78cc-pn982\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") " pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:00.992005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.991657    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": read tcp 10.134.0.2:33244->10.134.1.4:15021: read: connection reset by peer" start-of-body=
Oct 30 21:42:00.992005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:00.991733    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": read tcp 10.134.0.2:33244->10.134.1.4:15021: read: connection reset by peer"
Oct 30 21:42:01.076078 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:42:01.076038    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:42:01.161413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.160949    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:01.593236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.592922    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"]
Oct 30 21:42:01.593997 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:42:01.593942    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfee473a7_83af_4f47_bb06_fa7a9037e843.slice/crio-8003000ac3f1cd54a9abb032967d925d6ce845fd747ea00dc708953df2f5eefa WatchSource:0}: Error finding container 8003000ac3f1cd54a9abb032967d925d6ce845fd747ea00dc708953df2f5eefa: Status 404 returned error can't find the container with id 8003000ac3f1cd54a9abb032967d925d6ce845fd747ea00dc708953df2f5eefa
Oct 30 21:42:01.705298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.705266    6695 generic.go:356] "Generic (PLEG): container finished" podID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerID="4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713" exitCode=0
Oct 30 21:42:01.705485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.705342    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" event={"ID":"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb","Type":"ContainerDied","Data":"4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"}
Oct 30 21:42:01.705888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.705865    6695 scope.go:122] "RemoveContainer" containerID="4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"
Oct 30 21:42:01.707234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.707212    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982" event={"ID":"fee473a7-83af-4f47-bb06-fa7a9037e843","Type":"ContainerStarted","Data":"8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533"}
Oct 30 21:42:01.707333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.707248    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982" event={"ID":"fee473a7-83af-4f47-bb06-fa7a9037e843","Type":"ContainerStarted","Data":"8003000ac3f1cd54a9abb032967d925d6ce845fd747ea00dc708953df2f5eefa"}
Oct 30 21:42:01.707714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.707688    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:01.707771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.707755    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:01.707816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.707798    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:01.788935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:01.788876    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982" podStartSLOduration=1.788850433 podStartE2EDuration="1.788850433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:01.784115191 +0000 UTC m=+1904.706573492" watchObservedRunningTime="2026-10-30 21:42:01.788850433 +0000 UTC m=+1904.711308731"
Oct 30 21:42:02.162466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:02.162428    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:02.783681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:02.779158    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" event={"ID":"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb","Type":"ContainerStarted","Data":"b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c"}
Oct 30 21:42:03.431059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:03.431018    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:42:03.431763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:03.431672    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 30 21:42:03.431927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:03.431904    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 30 21:42:04.340627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:04.340136    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"]
Oct 30 21:42:04.351265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:04.351221    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"]
Oct 30 21:42:04.454125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:04.454074    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 30 21:42:04.454727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:04.454139    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 30 21:42:05.431627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:05.430946    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 30 21:42:05.431627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:05.431007    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 30 21:42:06.436030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.435990    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 30 21:42:06.436664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.436629    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 30 21:42:06.664131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.664093    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"]
Oct 30 21:42:06.669192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.669163    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:06.913628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.892267    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:42:06.913628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.892412    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:42:06.913628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.892814    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:42:06.999586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:06.999539    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:42:07.002502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.002467    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"]
Oct 30 21:42:07.042850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.039764    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnt9\" (UniqueName: \"kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.043229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.043201    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.043481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.043463    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.144760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.144704    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.145033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.145012    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnt9\" (UniqueName: \"kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.145114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.145063    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.145795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.145772    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.147329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.147297    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.176269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.176192    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnt9\" (UniqueName: \"kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9\") pod \"maas-api-revoke-sub-421b7397-9fxwf\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.345620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.345568    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:07.430696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.430126    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Oct 30 21:42:07.430696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.430201    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Oct 30 21:42:07.745582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:07.745548    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"]
Oct 30 21:42:07.747768 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:42:07.747742    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507fe9b0_408f_4ee8_b1c0_c55252a0e087.slice/crio-6257c05e788af7eacec3052b61fa42ae58cd0c3d57391129aadb0dc0cb2ecbb0 WatchSource:0}: Error finding container 6257c05e788af7eacec3052b61fa42ae58cd0c3d57391129aadb0dc0cb2ecbb0: Status 404 returned error can't find the container with id 6257c05e788af7eacec3052b61fa42ae58cd0c3d57391129aadb0dc0cb2ecbb0
Oct 30 21:42:08.060301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:08.060255    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf" event={"ID":"507fe9b0-408f-4ee8-b1c0-c55252a0e087","Type":"ContainerStarted","Data":"6257c05e788af7eacec3052b61fa42ae58cd0c3d57391129aadb0dc0cb2ecbb0"}
Oct 30 21:42:08.500584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:08.500479    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:42:08.501257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:08.501231    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:42:08.502822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:08.502798    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 21:42:09.071568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:09.071531    6695 generic.go:356] "Generic (PLEG): container finished" podID="507fe9b0-408f-4ee8-b1c0-c55252a0e087" containerID="2c2c56d6f20c0ed68893423c5da86212a161bfee177f8b9d0a6ba04fa12f6d95" exitCode=0
Oct 30 21:42:09.072004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:09.071648    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf" event={"ID":"507fe9b0-408f-4ee8-b1c0-c55252a0e087","Type":"ContainerDied","Data":"2c2c56d6f20c0ed68893423c5da86212a161bfee177f8b9d0a6ba04fa12f6d95"}
Oct 30 21:42:09.340547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:09.340500    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerName="payload-processing" containerID="cri-o://c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2" gracePeriod=25
Oct 30 21:42:09.352049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:09.352011    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing" containerID="cri-o://712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b" gracePeriod=25
Oct 30 21:42:09.512201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:09.512160    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:42:10.242569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.242538    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:10.304071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.304029    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9\" (UniqueName: \"kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9\") pod \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") "
Oct 30 21:42:10.304292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.304207    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca\") pod \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") "
Oct 30 21:42:10.304369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.304328    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca\") pod \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\" (UID: \"507fe9b0-408f-4ee8-b1c0-c55252a0e087\") "
Oct 30 21:42:10.304716 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.304681    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca" pod "507fe9b0-408f-4ee8-b1c0-c55252a0e087" (UID: "507fe9b0-408f-4ee8-b1c0-c55252a0e087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:10.305467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.305440    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:10.307036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.307002    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca" pod "507fe9b0-408f-4ee8-b1c0-c55252a0e087" (UID: "507fe9b0-408f-4ee8-b1c0-c55252a0e087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:10.307210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.307057    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9" pod "507fe9b0-408f-4ee8-b1c0-c55252a0e087" (UID: "507fe9b0-408f-4ee8-b1c0-c55252a0e087"). InnerVolumeSpecName "kube-api-access-pbnt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:10.407217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.407129    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbnt9\" (UniqueName: \"kubernetes.io/projected/507fe9b0-408f-4ee8-b1c0-c55252a0e087-kube-api-access-pbnt9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:10.407217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.407163    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507fe9b0-408f-4ee8-b1c0-c55252a0e087-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:10.519578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:10.519537    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:42:11.099120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:11.099071    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf" event={"ID":"507fe9b0-408f-4ee8-b1c0-c55252a0e087","Type":"ContainerDied","Data":"6257c05e788af7eacec3052b61fa42ae58cd0c3d57391129aadb0dc0cb2ecbb0"}
Oct 30 21:42:11.099120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:11.099116    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"
Oct 30 21:42:11.099417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:11.099177    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6257c05e788af7eacec3052b61fa42ae58cd0c3d57391129aadb0dc0cb2ecbb0"
Oct 30 21:42:11.161735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:11.161695    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:11.175211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:11.175175    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:12.121473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:12.121438    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:42:19.536232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:19.536165    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:42:20.067064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:20.067021    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"]
Oct 30 21:42:20.072160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:20.072099    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f94acbb7-hqhsm"]
Oct 30 21:42:20.536333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:20.536284    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:42:20.971749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:20.971663    6695 scope.go:122] "RemoveContainer" containerID="993ca4b8e70a02b38d62e78f0dcfd27e273b011be876476e846573d9f47fc7b5"
Oct 30 21:42:20.984570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:20.984535    6695 scope.go:122] "RemoveContainer" containerID="a5f544868ab2255f93ef401309ae98b7762f52f5188336806deadab083f7e205"
Oct 30 21:42:21.241903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:21.241799    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501b4020-c079-48c1-9396-8fb7815a7635" path="/var/lib/kubelet/pods/501b4020-c079-48c1-9396-8fb7815a7635/volumes"
Oct 30 21:42:27.356210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:27.356167    6695 generic.go:356] "Generic (PLEG): container finished" podID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerID="e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6" exitCode=0
Oct 30 21:42:27.357211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:27.357177    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" event={"ID":"1cb42d7c-2086-4352-a6b3-3e8f3c09da77","Type":"ContainerDied","Data":"e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"}
Oct 30 21:42:27.358060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:27.358014    6695 scope.go:122] "RemoveContainer" containerID="e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"
Oct 30 21:42:27.361667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:27.361590    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:27.361795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:27.361718    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:27.361795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:27.361756    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:28.368273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:28.368232    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" event={"ID":"1cb42d7c-2086-4352-a6b3-3e8f3c09da77","Type":"ContainerStarted","Data":"cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316"}
Oct 30 21:42:28.468954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:28.468920    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:42:28.469403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:28.469367    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:42:28.469625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:28.469582    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 21:42:29.456444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.456398    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:42:29.456963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.456472    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 21:42:29.538628 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:42:29.538352    6695 logging.go:55] [core] [Channel #3368 SubChannel #3369]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.225:50052", ServerName: "10.134.0.225:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.225:50052: connect: connection refused"
Oct 30 21:42:29.710202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.710168    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:42:29.773796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.773373    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl\" (UniqueName: \"kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl\") pod \"a7681615-c61a-41e6-90f3-c2132a3ed797\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") "
Oct 30 21:42:29.774398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.773430    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume\") pod \"a7681615-c61a-41e6-90f3-c2132a3ed797\" (UID: \"a7681615-c61a-41e6-90f3-c2132a3ed797\") "
Oct 30 21:42:29.774777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.774751    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume" pod "a7681615-c61a-41e6-90f3-c2132a3ed797" (UID: "a7681615-c61a-41e6-90f3-c2132a3ed797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:29.775483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.775463    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7681615-c61a-41e6-90f3-c2132a3ed797-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.778001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.777918    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl" pod "a7681615-c61a-41e6-90f3-c2132a3ed797" (UID: "a7681615-c61a-41e6-90f3-c2132a3ed797"). InnerVolumeSpecName "kube-api-access-lqhdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:29.782241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.782213    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:42:29.876575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.876539    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89\" (UniqueName: \"kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89\") pod \"9444f0d1-2133-48b7-8121-c44255fd2155\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") "
Oct 30 21:42:29.876575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.876584    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume\") pod \"9444f0d1-2133-48b7-8121-c44255fd2155\" (UID: \"9444f0d1-2133-48b7-8121-c44255fd2155\") "
Oct 30 21:42:29.877170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.877147    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhdl\" (UniqueName: \"kubernetes.io/projected/a7681615-c61a-41e6-90f3-c2132a3ed797-kube-api-access-lqhdl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.877499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.877478    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume" pod "9444f0d1-2133-48b7-8121-c44255fd2155" (UID: "9444f0d1-2133-48b7-8121-c44255fd2155"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:29.879654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.879627    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89" pod "9444f0d1-2133-48b7-8121-c44255fd2155" (UID: "9444f0d1-2133-48b7-8121-c44255fd2155"). InnerVolumeSpecName "kube-api-access-8ks89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:29.997627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.997358    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ks89\" (UniqueName: \"kubernetes.io/projected/9444f0d1-2133-48b7-8121-c44255fd2155-kube-api-access-8ks89\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:29.997627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:29.997396    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9444f0d1-2133-48b7-8121-c44255fd2155-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:42:30.391418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.391386    6695 generic.go:356] "Generic (PLEG): container finished" podID="9444f0d1-2133-48b7-8121-c44255fd2155" containerID="712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b" exitCode=0
Oct 30 21:42:30.391653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.391442    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:42:30.391653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.391469    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" event={"ID":"9444f0d1-2133-48b7-8121-c44255fd2155","Type":"ContainerDied","Data":"712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"}
Oct 30 21:42:30.391653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.391524    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" event={"ID":"9444f0d1-2133-48b7-8121-c44255fd2155","Type":"ContainerDied","Data":"a02c60cf54664708d92aad0c632fbdbcd3396eef1f71554d2391a76eb79906fb"}
Oct 30 21:42:30.391653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.391547    6695 scope.go:122] "RemoveContainer" containerID="712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"
Oct 30 21:42:30.393475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.393450    6695 generic.go:356] "Generic (PLEG): container finished" podID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerID="c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2" exitCode=0
Oct 30 21:42:30.393645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.393503    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"
Oct 30 21:42:30.393645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.393504    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" event={"ID":"a7681615-c61a-41e6-90f3-c2132a3ed797","Type":"ContainerDied","Data":"c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2"}
Oct 30 21:42:30.393645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.393625    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws" event={"ID":"a7681615-c61a-41e6-90f3-c2132a3ed797","Type":"ContainerDied","Data":"84893528202fe04b347ea4abeb38e67784aa3271f9268ba1cb90b0fd4a55a8dd"}
Oct 30 21:42:30.404315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.404295    6695 scope.go:122] "RemoveContainer" containerID="712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"
Oct 30 21:42:30.404680 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:42:30.404657    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b\": container with ID starting with 712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b not found: ID does not exist" containerID="712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"
Oct 30 21:42:30.404784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.404692    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"} err="failed to get container status \"712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b\": rpc error: code = NotFound desc = could not find container \"712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b\": container with ID starting with 712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b not found: ID does not exist"
Oct 30 21:42:30.404784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.404715    6695 scope.go:122] "RemoveContainer" containerID="c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2"
Oct 30 21:42:30.465629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.465061    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:42:30.465629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.465132    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 21:42:30.493633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.485671    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"]
Oct 30 21:42:30.493633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.489527    6695 scope.go:122] "RemoveContainer" containerID="c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2"
Oct 30 21:42:30.493633 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:42:30.490075    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2\": container with ID starting with c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2 not found: ID does not exist" containerID="c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2"
Oct 30 21:42:30.493633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.490128    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2"} err="failed to get container status \"c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2\": rpc error: code = NotFound desc = could not find container \"c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2\": container with ID starting with c78c0819299f5e817452c477bb53417021b761ece569488b52e77ff17eb304b2 not found: ID does not exist"
Oct 30 21:42:30.493633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.492080    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c19b75-58ddf4844f-5xgws"]
Oct 30 21:42:30.556962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.556914    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.225:50052\" within 1s: context deadline exceeded"
Oct 30 21:42:30.557284 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:42:30.556974    6695 logging.go:55] [core] [Channel #3368 SubChannel #3369]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.225:50052", ServerName: "10.134.0.225:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.225:50052: operation was canceled"
Oct 30 21:42:30.557733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.557694    6695 status_manager.go:453] "Container readiness changed before pod has synced" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh" containerID="cri-o://712591c24e810bb707a6ae6191c75b373caaabadce82e5a1007559dc32923e6b"
Oct 30 21:42:30.557916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.557902    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"
Oct 30 21:42:30.560223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.559803    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"]
Oct 30 21:42:30.564363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:30.564327    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c19b75-559d85bb7-klnmh"]
Oct 30 21:42:31.078400 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:42:31.078360    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:42:31.254543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:31.254503    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" path="/var/lib/kubelet/pods/9444f0d1-2133-48b7-8121-c44255fd2155/volumes"
Oct 30 21:42:31.255631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:31.255589    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" path="/var/lib/kubelet/pods/a7681615-c61a-41e6-90f3-c2132a3ed797/volumes"
Oct 30 21:42:31.464182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:31.463095    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:42:31.464182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:31.463176    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 21:42:31.854709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:31.854675    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:42:32.456555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:32.456515    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:42:32.456788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:32.456594    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 21:42:33.456430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:33.456393    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:42:33.456989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:33.456470    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Oct 30 21:42:34.488993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.487253    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:42:34.491156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.491001    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 21:42:34.828466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.828430    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"]
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829534    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerName="payload-processing"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829556    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerName="payload-processing"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829619    6695 cpu_manager.go:414] "containerMap: removing container" podUID="507fe9b0-408f-4ee8-b1c0-c55252a0e087" containerName="revoke-keys"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829628    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507fe9b0-408f-4ee8-b1c0-c55252a0e087" containerName="revoke-keys"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829676    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829684    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829855    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9444f0d1-2133-48b7-8121-c44255fd2155" containerName="payload-pre-processing"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829878    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7681615-c61a-41e6-90f3-c2132a3ed797" containerName="payload-processing"
Oct 30 21:42:34.829929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.829893    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="507fe9b0-408f-4ee8-b1c0-c55252a0e087" containerName="revoke-keys"
Oct 30 21:42:34.835946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.835920    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:34.855169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.855137    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-5494e8\"/\"kube-root-ca.crt\""
Oct 30 21:42:34.860794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.860520    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-5494e8\"/\"openshift-service-ca.crt\""
Oct 30 21:42:34.860794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.860623    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-5494e8\"/\"auth-test-model-5494e8-kserve-self-signed-certs\""
Oct 30 21:42:34.860794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.860702    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-5494e8\"/\"default-dockercfg-25ngl\""
Oct 30 21:42:34.864518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.864488    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"]
Oct 30 21:42:34.883739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.883678    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"]
Oct 30 21:42:34.893478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.893434    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c19b75-vj4vh"]
Oct 30 21:42:34.955352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.955315    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:34.955539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.955459    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:34.955613 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.955579    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:34.955720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.955702    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:34.955774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.955759    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg89k\" (UniqueName: \"kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:34.955914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:34.955884    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057293    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057369    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057414    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057445    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057478    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg89k\" (UniqueName: \"kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057521    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057843    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.057881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057866    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.058011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.057950    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.060407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.060236    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.061406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.061377    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.109847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.109771    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg89k\" (UniqueName: \"kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k\") pod \"auth-test-model-5494e8-kserve-567cc45f45-4p29r\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") " pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.173641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.171491    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:35.270685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.268330    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f86b536-58f7-4170-a836-12ebecc66f4f" path="/var/lib/kubelet/pods/4f86b536-58f7-4170-a836-12ebecc66f4f/volumes"
Oct 30 21:42:35.607913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:35.607887    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"]
Oct 30 21:42:35.610021 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:42:35.609992    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b260740_d39b_484c_b3b9_3c82910f90e4.slice/crio-82d08319f8cb01893d680361be95ef382c9783acaabcceb43af95d1c1ab058a6 WatchSource:0}: Error finding container 82d08319f8cb01893d680361be95ef382c9783acaabcceb43af95d1c1ab058a6: Status 404 returned error can't find the container with id 82d08319f8cb01893d680361be95ef382c9783acaabcceb43af95d1c1ab058a6
Oct 30 21:42:36.551886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:36.551838    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" event={"ID":"1b260740-d39b-484c-b3b9-3c82910f90e4","Type":"ContainerStarted","Data":"615797fb7170449c0c048455df82e54f5b84654f3f2fb3e47a8f918177102fc8"}
Oct 30 21:42:36.552078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:36.551904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" event={"ID":"1b260740-d39b-484c-b3b9-3c82910f90e4","Type":"ContainerStarted","Data":"82d08319f8cb01893d680361be95ef382c9783acaabcceb43af95d1c1ab058a6"}
Oct 30 21:42:36.624928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:36.624873    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" podStartSLOduration=2.624847662 podStartE2EDuration="2.624847662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:36.623575066 +0000 UTC m=+1939.546033365" watchObservedRunningTime="2026-10-30 21:42:36.624847662 +0000 UTC m=+1939.547305959"
Oct 30 21:42:37.172441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:37.172401    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:44.766092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.766055    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"]
Oct 30 21:42:44.771592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.771565    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.808672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.803230    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"]
Oct 30 21:42:44.808672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.806875    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-bfabe1\"/\"auth-test-model-bfabe1-kserve-self-signed-certs\""
Oct 30 21:42:44.808672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.806994    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-bfabe1\"/\"kube-root-ca.crt\""
Oct 30 21:42:44.808672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.807073    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-bfabe1\"/\"openshift-service-ca.crt\""
Oct 30 21:42:44.808672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.807253    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-bfabe1\"/\"default-dockercfg-x6fkh\""
Oct 30 21:42:44.842222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.842181    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.842437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.842252    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.842437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.842280    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.843241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.842477    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.843241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.842532    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.843241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.842626    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcsw\" (UniqueName: \"kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.943922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.943881    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.943961    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcsw\" (UniqueName: \"kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944074    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944110    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944128    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944263    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944547    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944576    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.944834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.944800    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.946572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.946548    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.946769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.946749    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:44.963150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:44.963115    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcsw\" (UniqueName: \"kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw\") pod \"auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") " pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:45.106791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:45.106755    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:45.175629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:45.174970    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:45.422668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:45.422231    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:45.696150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:45.696059    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:42:45.701621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:45.701582    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"]
Oct 30 21:42:46.692740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:46.692695    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" event={"ID":"2d0be272-d582-4a67-8956-c820a5ed5ed4","Type":"ContainerStarted","Data":"42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb"}
Oct 30 21:42:46.692740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:46.692744    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" event={"ID":"2d0be272-d582-4a67-8956-c820a5ed5ed4","Type":"ContainerStarted","Data":"5de3b5e35a43421b4bded1eb5eeb6bef0e0c61d6f705da1cbaebd7ea52eeae6d"}
Oct 30 21:42:46.735017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:46.734949    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" podStartSLOduration=2.734930538 podStartE2EDuration="2.734930538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:46.729912964 +0000 UTC m=+1949.652371263" watchObservedRunningTime="2026-10-30 21:42:46.734930538 +0000 UTC m=+1949.657388835"
Oct 30 21:42:47.106911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:47.106877    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:55.106951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:55.106913    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:55.121226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:55.121189    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:42:55.862886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:42:55.862847    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:43:01.075208 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:43:01.075160    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:43:10.033802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:10.033744    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"]
Oct 30 21:43:10.045164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:10.045118    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e6ac19-jtszc"]
Oct 30 21:43:11.216906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:11.216870    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75866545-81f3-4335-abf8-3a2941aaaf06" path="/var/lib/kubelet/pods/75866545-81f3-4335-abf8-3a2941aaaf06/volumes"
Oct 30 21:43:21.055221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:21.055184    6695 scope.go:122] "RemoveContainer" containerID="2561293cfa707f243dee3f1b361de947178bc072e62e3fb06430c7bc44a66573"
Oct 30 21:43:21.066452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:21.066428    6695 scope.go:122] "RemoveContainer" containerID="5893cd34e8117d3241d91ec04618fbcb1ddfbeb7acc973c6c58ea0458fc3fa81"
Oct 30 21:43:21.077127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:21.077099    6695 scope.go:122] "RemoveContainer" containerID="2d6c8d00030c85aa3e9a2d70fc7d5cfc330bbd2339db8341776f9f644ef86af6"
Oct 30 21:43:24.124266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.124223    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"]
Oct 30 21:43:24.131623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.131579    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.158899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.158856    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-ae6492\"/\"default-dockercfg-g8g9r\""
Oct 30 21:43:24.158899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.158889    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-ae6492\"/\"kube-root-ca.crt\""
Oct 30 21:43:24.159246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.158858    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-ae6492\"/\"openshift-service-ca.crt\""
Oct 30 21:43:24.159246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.158984    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-ae6492\"/\"auth-test-model-ae6492-kserve-self-signed-certs\""
Oct 30 21:43:24.162839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.162795    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.163102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.163022    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.163102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.163048    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.163231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.163187    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.163231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.163221    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.163366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.163351    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.165787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.165736    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"]
Oct 30 21:43:24.264711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.264591    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.264921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.264727    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.264921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.264886    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.265516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.264960    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.265516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.265103    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.265516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.265116    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.265516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.265203    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.265516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.265209    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.265516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.265478    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.267240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.267215    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.267677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.267658    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.285546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.285505    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t\") pod \"auth-test-model-ae6492-kserve-865d4bffc7-9lvw2\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") " pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.460689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.454001    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:24.862461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:24.862427    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"]
Oct 30 21:43:24.864699 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:43:24.864662    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c5677f9_2262_49ac_9694_b71b2398cc88.slice/crio-f40f8ab44162e79c960ab7dce49716c6c89bb499dcbe5e8871af15eb9cf0dac5 WatchSource:0}: Error finding container f40f8ab44162e79c960ab7dce49716c6c89bb499dcbe5e8871af15eb9cf0dac5: Status 404 returned error can't find the container with id f40f8ab44162e79c960ab7dce49716c6c89bb499dcbe5e8871af15eb9cf0dac5
Oct 30 21:43:25.272870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:25.272590    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" event={"ID":"7c5677f9-2262-49ac-9694-b71b2398cc88","Type":"ContainerStarted","Data":"9d27d3a90bb5ba31fdc36e3c18ad209ddb08ab76208eae6e132f9b59969e68eb"}
Oct 30 21:43:25.273344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:25.272886    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" event={"ID":"7c5677f9-2262-49ac-9694-b71b2398cc88","Type":"ContainerStarted","Data":"f40f8ab44162e79c960ab7dce49716c6c89bb499dcbe5e8871af15eb9cf0dac5"}
Oct 30 21:43:25.326067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:25.326004    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" podStartSLOduration=1.32598726 podStartE2EDuration="1.32598726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:25.32081871 +0000 UTC m=+1988.243277009" watchObservedRunningTime="2026-10-30 21:43:25.32598726 +0000 UTC m=+1988.248445557"
Oct 30 21:43:25.485746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:25.485707    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:25.841770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:25.841728    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"]
Oct 30 21:43:25.842023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:25.841987    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" containerID="cri-o://e12294f73bafb48b38deb1cf468166714d14ddd9d53073b7a5c5306df0241ffc" gracePeriod=30
Oct 30 21:43:26.451788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.451745    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"]
Oct 30 21:43:26.456994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.456960    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.477054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.477016    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"]
Oct 30 21:43:26.482450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.482410    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:43:26.482632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.482547    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:43:26.482632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.482549    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:43:26.482632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.482555    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:43:26.533486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.533441    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.533762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.533597    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.533884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.533859    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8zb\" (UniqueName: \"kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.635521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.635478    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8zb\" (UniqueName: \"kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.635732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.635673    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.635779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.635742    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.636408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.636380    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.638289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.638259    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.654912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.654881    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8zb\" (UniqueName: \"kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb\") pod \"maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:26.773971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:26.773926    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:27.034932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:27.034827    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"]
Oct 30 21:43:27.039687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:27.039646    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d6d355-zr7q4"]
Oct 30 21:43:27.090174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:27.090140    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"]
Oct 30 21:43:27.092344 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:43:27.092302    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe732643_f8b5_4c64_922a_a007b1b0854a.slice/crio-65ba15c02b913a334424f699b0da2d3febb873136172db828c1ddb5cbbc534a0 WatchSource:0}: Error finding container 65ba15c02b913a334424f699b0da2d3febb873136172db828c1ddb5cbbc534a0: Status 404 returned error can't find the container with id 65ba15c02b913a334424f699b0da2d3febb873136172db828c1ddb5cbbc534a0
Oct 30 21:43:27.217284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:27.217246    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb07dd3-b043-4400-ac5d-df0c69b3d2eb" path="/var/lib/kubelet/pods/7bb07dd3-b043-4400-ac5d-df0c69b3d2eb/volumes"
Oct 30 21:43:27.294414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:27.294308    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c" event={"ID":"be732643-f8b5-4c64-922a-a007b1b0854a","Type":"ContainerStarted","Data":"65ba15c02b913a334424f699b0da2d3febb873136172db828c1ddb5cbbc534a0"}
Oct 30 21:43:28.304895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:28.304860    6695 generic.go:356] "Generic (PLEG): container finished" podID="be732643-f8b5-4c64-922a-a007b1b0854a" containerID="688c4da4de7ed1493e06585fb816073021fd6e16cf131d66a799fc4743760117" exitCode=0
Oct 30 21:43:28.305305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:28.304950    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c" event={"ID":"be732643-f8b5-4c64-922a-a007b1b0854a","Type":"ContainerDied","Data":"688c4da4de7ed1493e06585fb816073021fd6e16cf131d66a799fc4743760117"}
Oct 30 21:43:29.499885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.499851    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:29.594679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.594535    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb\" (UniqueName: \"kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb\") pod \"be732643-f8b5-4c64-922a-a007b1b0854a\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") "
Oct 30 21:43:29.594882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.594839    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca\") pod \"be732643-f8b5-4c64-922a-a007b1b0854a\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") "
Oct 30 21:43:29.595008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.594987    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca\") pod \"be732643-f8b5-4c64-922a-a007b1b0854a\" (UID: \"be732643-f8b5-4c64-922a-a007b1b0854a\") "
Oct 30 21:43:29.595871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.595572    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca" pod "be732643-f8b5-4c64-922a-a007b1b0854a" (UID: "be732643-f8b5-4c64-922a-a007b1b0854a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:29.595871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.595839    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:29.600651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.597527    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca" pod "be732643-f8b5-4c64-922a-a007b1b0854a" (UID: "be732643-f8b5-4c64-922a-a007b1b0854a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:29.600651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.597699    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb" pod "be732643-f8b5-4c64-922a-a007b1b0854a" (UID: "be732643-f8b5-4c64-922a-a007b1b0854a"). InnerVolumeSpecName "kube-api-access-qd8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:29.697808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.697769    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be732643-f8b5-4c64-922a-a007b1b0854a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:29.697808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.697805    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd8zb\" (UniqueName: \"kubernetes.io/projected/be732643-f8b5-4c64-922a-a007b1b0854a-kube-api-access-qd8zb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:29.963067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.962977    6695 patch_prober.go:29] interesting pod/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:43:29.963067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:29.963040    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:43:30.326478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:30.326438    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"
Oct 30 21:43:30.326478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:30.326459    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c" event={"ID":"be732643-f8b5-4c64-922a-a007b1b0854a","Type":"ContainerDied","Data":"65ba15c02b913a334424f699b0da2d3febb873136172db828c1ddb5cbbc534a0"}
Oct 30 21:43:30.326823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:30.326517    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ba15c02b913a334424f699b0da2d3febb873136172db828c1ddb5cbbc534a0"
Oct 30 21:43:31.260088 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:43:31.208848    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:43:31.405828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.405785    6695 generic.go:356] "Generic (PLEG): container finished" podID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerID="e12294f73bafb48b38deb1cf468166714d14ddd9d53073b7a5c5306df0241ffc" exitCode=0
Oct 30 21:43:31.406029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.406013    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" event={"ID":"f10ff2df-7e2d-4273-b248-250f6f73d33e","Type":"ContainerDied","Data":"e12294f73bafb48b38deb1cf468166714d14ddd9d53073b7a5c5306df0241ffc"}
Oct 30 21:43:31.406129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.406072    6695 scope.go:122] "RemoveContainer" containerID="cce57ec403c035183799d0b9483f0a91143fbe077056811bedd8f2ee99e3090d"
Oct 30 21:43:31.449445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.449419    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:43:31.565133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565099    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.565133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565147    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565189    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565329    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565452    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565495    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565549    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565685    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.565778    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert\") pod \"f10ff2df-7e2d-4273-b248-250f6f73d33e\" (UID: \"f10ff2df-7e2d-4273-b248-250f6f73d33e\") "
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.566961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.567164    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:31.567819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.567593    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:31.568422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.567886    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:31.568422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.568313    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:31.568849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.568821    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:31.570031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.570000    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:43:31.570737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.570696    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "kube-api-access-rvcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:31.570925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.570792    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token" pod "f10ff2df-7e2d-4273-b248-250f6f73d33e" (UID: "f10ff2df-7e2d-4273-b248-250f6f73d33e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:31.667563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667520    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667557    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667567    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvcdt\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-kube-api-access-rvcdt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667596    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667622    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10ff2df-7e2d-4273-b248-250f6f73d33e-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667634    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667649    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667661    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10ff2df-7e2d-4273-b248-250f6f73d33e-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:31.667830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:31.667673    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10ff2df-7e2d-4273-b248-250f6f73d33e-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:32.417002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:32.416970    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"
Oct 30 21:43:32.417485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:32.417001    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842" event={"ID":"f10ff2df-7e2d-4273-b248-250f6f73d33e","Type":"ContainerDied","Data":"80ef3f16f69003111480e1de11923a83a21afddff2ec164cfe958f6ea65c5f05"}
Oct 30 21:43:32.417485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:32.417068    6695 scope.go:122] "RemoveContainer" containerID="e12294f73bafb48b38deb1cf468166714d14ddd9d53073b7a5c5306df0241ffc"
Oct 30 21:43:32.516176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:32.516130    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"]
Oct 30 21:43:32.521556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:32.521516    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c19b75-openshift-default-d867fdd97-5h842"]
Oct 30 21:43:33.218140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:33.218101    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" path="/var/lib/kubelet/pods/f10ff2df-7e2d-4273-b248-250f6f73d33e/volumes"
Oct 30 21:43:34.454929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:34.454882    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:34.468278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:34.468247    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:35.045942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:35.045892    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"]
Oct 30 21:43:35.050918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:35.050875    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0999ea2-6fhhh"]
Oct 30 21:43:35.216718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:35.216680    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d495e107-386e-47ec-8cf6-6efaa3c6a5c3" path="/var/lib/kubelet/pods/d495e107-386e-47ec-8cf6-6efaa3c6a5c3/volumes"
Oct 30 21:43:35.509882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:35.509788    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:43:37.004001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:37.003954    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"]
Oct 30 21:43:37.039116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:37.039077    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"]
Oct 30 21:43:38.835628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:38.829914    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"]
Oct 30 21:43:38.852076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:38.852006    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7a4de0-kjz9c"]
Oct 30 21:43:39.035482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:39.035443    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"]
Oct 30 21:43:39.043917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:39.043851    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-557e1bee-vcx8d"]
Oct 30 21:43:39.217235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:39.217146    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751c146a-33a9-4f3b-b924-3d6ad36e32ad" path="/var/lib/kubelet/pods/751c146a-33a9-4f3b-b924-3d6ad36e32ad/volumes"
Oct 30 21:43:39.217818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:39.217795    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be732643-f8b5-4c64-922a-a007b1b0854a" path="/var/lib/kubelet/pods/be732643-f8b5-4c64-922a-a007b1b0854a/volumes"
Oct 30 21:43:42.004962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.004911    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7" podUID="cb8fafcf-bb27-46a5-af58-bac88561bf62" containerName="payload-processing" containerID="cri-o://c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b" gracePeriod=25
Oct 30 21:43:42.040481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.040436    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j" podUID="20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" containerName="payload-pre-processing" containerID="cri-o://2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751" gracePeriod=25
Oct 30 21:43:42.333026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.332994    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:43:42.378662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.378633    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:43:42.452249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.452206    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume\") pod \"cb8fafcf-bb27-46a5-af58-bac88561bf62\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") "
Oct 30 21:43:42.452743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.452712    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2\" (UniqueName: \"kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2\") pod \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") "
Oct 30 21:43:42.453018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.452971    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2\" (UniqueName: \"kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2\") pod \"cb8fafcf-bb27-46a5-af58-bac88561bf62\" (UID: \"cb8fafcf-bb27-46a5-af58-bac88561bf62\") "
Oct 30 21:43:42.453149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.453116    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume\") pod \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\" (UID: \"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887\") "
Oct 30 21:43:42.454793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.454747    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume" pod "20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" (UID: "20e0ebfe-5a5b-42d7-ae03-7cc476ef3887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:42.456549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.456274    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2" pod "20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" (UID: "20e0ebfe-5a5b-42d7-ae03-7cc476ef3887"). InnerVolumeSpecName "kube-api-access-8ltp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:42.457459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.456959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume" pod "cb8fafcf-bb27-46a5-af58-bac88561bf62" (UID: "cb8fafcf-bb27-46a5-af58-bac88561bf62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:42.457459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.457314    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2" pod "cb8fafcf-bb27-46a5-af58-bac88561bf62" (UID: "cb8fafcf-bb27-46a5-af58-bac88561bf62"). InnerVolumeSpecName "kube-api-access-5tgp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:42.555382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.555336    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltp2\" (UniqueName: \"kubernetes.io/projected/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-kube-api-access-8ltp2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:42.555382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.555379    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tgp2\" (UniqueName: \"kubernetes.io/projected/cb8fafcf-bb27-46a5-af58-bac88561bf62-kube-api-access-5tgp2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:42.555382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.555391    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:42.555707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.555401    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fafcf-bb27-46a5-af58-bac88561bf62-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:42.905498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.905401    6695 generic.go:356] "Generic (PLEG): container finished" podID="cb8fafcf-bb27-46a5-af58-bac88561bf62" containerID="c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b" exitCode=0
Oct 30 21:43:42.905498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.905479    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7" event={"ID":"cb8fafcf-bb27-46a5-af58-bac88561bf62","Type":"ContainerDied","Data":"c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b"}
Oct 30 21:43:42.905789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.905492    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"
Oct 30 21:43:42.905789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.905571    6695 scope.go:122] "RemoveContainer" containerID="c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b"
Oct 30 21:43:42.905789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.905561    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7" event={"ID":"cb8fafcf-bb27-46a5-af58-bac88561bf62","Type":"ContainerDied","Data":"935fb27b8ec2e9f07dab9e4312012c20c416bd9409e8a2e0f308cff5ec6853a3"}
Oct 30 21:43:42.907108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.907077    6695 generic.go:356] "Generic (PLEG): container finished" podID="20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" containerID="2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751" exitCode=0
Oct 30 21:43:42.907216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.907136    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j" event={"ID":"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887","Type":"ContainerDied","Data":"2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751"}
Oct 30 21:43:42.907216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.907148    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"
Oct 30 21:43:42.907216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.907175    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j" event={"ID":"20e0ebfe-5a5b-42d7-ae03-7cc476ef3887","Type":"ContainerDied","Data":"8fd822efb1ed64901ff87a7d118490b44fe089d478768c1e6f62c0f6afbe818f"}
Oct 30 21:43:42.920413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.920386    6695 scope.go:122] "RemoveContainer" containerID="c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b"
Oct 30 21:43:42.920784 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:43:42.920757    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b\": container with ID starting with c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b not found: ID does not exist" containerID="c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b"
Oct 30 21:43:42.920887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.920798    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b"} err="failed to get container status \"c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b\": rpc error: code = NotFound desc = could not find container \"c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b\": container with ID starting with c5e699c327c65c34ee8801e41a2d29954dfc7c46915a27d64d04dd6659b4503b not found: ID does not exist"
Oct 30 21:43:42.920887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.920823    6695 scope.go:122] "RemoveContainer" containerID="2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751"
Oct 30 21:43:42.932551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.932527    6695 scope.go:122] "RemoveContainer" containerID="2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751"
Oct 30 21:43:42.932894 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:43:42.932872    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751\": container with ID starting with 2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751 not found: ID does not exist" containerID="2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751"
Oct 30 21:43:42.932963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.932901    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751"} err="failed to get container status \"2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751\": rpc error: code = NotFound desc = could not find container \"2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751\": container with ID starting with 2fdc1d5296d347c2ec6db2a0bd485a68f7e703722b3e1b0cfb15a91bdc156751 not found: ID does not exist"
Oct 30 21:43:42.966555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.966518    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"]
Oct 30 21:43:42.978148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:42.978114    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7a4de0-6876496559-4qd2j"]
Oct 30 21:43:43.008914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:43.008875    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"]
Oct 30 21:43:43.017169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:43.017132    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7a4de0-5dd79d7987-hdxj7"]
Oct 30 21:43:43.259584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:43.259487    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" path="/var/lib/kubelet/pods/20e0ebfe-5a5b-42d7-ae03-7cc476ef3887/volumes"
Oct 30 21:43:43.260256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:43.260230    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8fafcf-bb27-46a5-af58-bac88561bf62" path="/var/lib/kubelet/pods/cb8fafcf-bb27-46a5-af58-bac88561bf62/volumes"
Oct 30 21:43:47.256768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:47.256726    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"]
Oct 30 21:43:47.257283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:47.257056    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy" containerID="cri-o://a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0" gracePeriod=30
Oct 30 21:43:48.044300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:48.044259    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"]
Oct 30 21:43:48.053354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:48.053303    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccc0bb62-x8rs4"]
Oct 30 21:43:49.044241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:49.044208    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"]
Oct 30 21:43:49.056727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:49.056696    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef4b4acf-chzvf"]
Oct 30 21:43:49.215429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:49.215394    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cbfa8f-350c-4a6a-9320-59c5959ae58a" path="/var/lib/kubelet/pods/91cbfa8f-350c-4a6a-9320-59c5959ae58a/volumes"
Oct 30 21:43:49.215933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:49.215918    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b493d137-4288-4e93-900b-715f3771da7f" path="/var/lib/kubelet/pods/b493d137-4288-4e93-900b-715f3771da7f/volumes"
Oct 30 21:43:50.513190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.513143    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"]
Oct 30 21:43:50.514147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514123    6695 cpu_manager.go:414] "containerMap: removing container" podUID="20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" containerName="payload-pre-processing"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514147    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" containerName="payload-pre-processing"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514185    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514193    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514209    6695 cpu_manager.go:414] "containerMap: removing container" podUID="be732643-f8b5-4c64-922a-a007b1b0854a" containerName="revoke-keys"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514216    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be732643-f8b5-4c64-922a-a007b1b0854a" containerName="revoke-keys"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514269    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy"
Oct 30 21:43:50.514285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514276    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy"
Oct 30 21:43:50.514712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514290    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cb8fafcf-bb27-46a5-af58-bac88561bf62" containerName="payload-processing"
Oct 30 21:43:50.514712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514299    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8fafcf-bb27-46a5-af58-bac88561bf62" containerName="payload-processing"
Oct 30 21:43:50.514712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514482    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy"
Oct 30 21:43:50.514712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514498    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="be732643-f8b5-4c64-922a-a007b1b0854a" containerName="revoke-keys"
Oct 30 21:43:50.514712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514519    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8fafcf-bb27-46a5-af58-bac88561bf62" containerName="payload-processing"
Oct 30 21:43:50.514712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.514532    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e0ebfe-5a5b-42d7-ae03-7cc476ef3887" containerName="payload-pre-processing"
Oct 30 21:43:50.520584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.520555    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.545273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.545237    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"]
Oct 30 21:43:50.548144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.548122    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-64d142\"/\"openshift-service-ca.crt\""
Oct 30 21:43:50.548480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.548456    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-64d142\"/\"kube-root-ca.crt\""
Oct 30 21:43:50.557325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.557296    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-64d142\"/\"auth-test-model-64d142-kserve-self-signed-certs\""
Oct 30 21:43:50.557505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.557324    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-64d142\"/\"default-dockercfg-k5d7p\""
Oct 30 21:43:50.596095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.596057    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.596323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.596147    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.596323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.596234    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.596323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.596280    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.596509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.596494    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmfh\" (UniqueName: \"kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.596549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.596533    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698014    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698093    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698310    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmfh\" (UniqueName: \"kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698366    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698410    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698470    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698763    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.698869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.698844    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.700456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.700433    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.700931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.700908    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.719432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.719397    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmfh\" (UniqueName: \"kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh\") pod \"auth-test-model-64d142-kserve-574d444489-fq9kf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") " pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:50.845012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:50.844374    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:51.270953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:51.270920    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"]
Oct 30 21:43:51.271901 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:43:51.271872    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fbef89e_d921_4885_8cd1_850bcaf697cf.slice/crio-9fbf5451dc55b46e66ec2693f917fa240ec0017064faaf2a86afcaf0d9a290b5 WatchSource:0}: Error finding container 9fbf5451dc55b46e66ec2693f917fa240ec0017064faaf2a86afcaf0d9a290b5: Status 404 returned error can't find the container with id 9fbf5451dc55b46e66ec2693f917fa240ec0017064faaf2a86afcaf0d9a290b5
Oct 30 21:43:52.006054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.006011    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" event={"ID":"1fbef89e-d921-4885-8cd1-850bcaf697cf","Type":"ContainerStarted","Data":"d5920e4d825304908c90d7bb2bcf115b38a2be5f95de2652160975d4c272af2c"}
Oct 30 21:43:52.006457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.006063    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" event={"ID":"1fbef89e-d921-4885-8cd1-850bcaf697cf","Type":"ContainerStarted","Data":"9fbf5451dc55b46e66ec2693f917fa240ec0017064faaf2a86afcaf0d9a290b5"}
Oct 30 21:43:52.039507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.039463    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"]
Oct 30 21:43:52.048580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.048541    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d4454b-5fw5k"]
Oct 30 21:43:52.050849 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.050804    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" podStartSLOduration=2.050791223 podStartE2EDuration="2.050791223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:52.047095487 +0000 UTC m=+2014.969553785" watchObservedRunningTime="2026-10-30 21:43:52.050791223 +0000 UTC m=+2014.973249520"
Oct 30 21:43:52.533049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.532986    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:43:52.550934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.550900    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.550962    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551044    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551086    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551130    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551161    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551238    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551283    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551307    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:52.551345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551328    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token\") pod \"2c7f4209-3fba-41a2-b098-b81984be14cf\" (UID: \"2c7f4209-3fba-41a2-b098-b81984be14cf\") "
Oct 30 21:43:52.551659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551634    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:52.551659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551646    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:52.551816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551655    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:52.551883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.551851    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:52.552244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.552216    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.552244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.552243    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.552403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.552254    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.552403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.552267    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7f4209-3fba-41a2-b098-b81984be14cf-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.552403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.552280    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.553448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.553422    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:52.553728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.553705    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:43:52.553809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.553746    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "kube-api-access-6474g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:52.553877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.553860    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token" pod "2c7f4209-3fba-41a2-b098-b81984be14cf" (UID: "2c7f4209-3fba-41a2-b098-b81984be14cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:52.654211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.654160    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.654211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.654210    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.654450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.654225    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7f4209-3fba-41a2-b098-b81984be14cf-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.654450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.654242    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6474g\" (UniqueName: \"kubernetes.io/projected/2c7f4209-3fba-41a2-b098-b81984be14cf-kube-api-access-6474g\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:43:52.845673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:52.845582    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:43:53.017365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.017309    6695 generic.go:356] "Generic (PLEG): container finished" podID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerID="a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0" exitCode=0
Oct 30 21:43:53.017365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.017363    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"
Oct 30 21:43:53.017894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.017390    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" event={"ID":"2c7f4209-3fba-41a2-b098-b81984be14cf","Type":"ContainerDied","Data":"a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0"}
Oct 30 21:43:53.017894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.017455    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm" event={"ID":"2c7f4209-3fba-41a2-b098-b81984be14cf","Type":"ContainerDied","Data":"9750eaaaf96dc9b63bdd58a03b7e482ac2f02d001a865d757ece3e62bc6669d0"}
Oct 30 21:43:53.017894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.017478    6695 scope.go:122] "RemoveContainer" containerID="a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0"
Oct 30 21:43:53.030996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.030968    6695 scope.go:122] "RemoveContainer" containerID="2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"
Oct 30 21:43:53.042894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.042848    6695 scope.go:122] "RemoveContainer" containerID="a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0"
Oct 30 21:43:53.043330 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:43:53.043304    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0\": container with ID starting with a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0 not found: ID does not exist" containerID="a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0"
Oct 30 21:43:53.043457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.043339    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0"} err="failed to get container status \"a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0\": rpc error: code = NotFound desc = could not find container \"a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0\": container with ID starting with a86df4cca01a02b5e11e5731f5e117864762cee78d2bb58bfdc224f0203322c0 not found: ID does not exist"
Oct 30 21:43:53.043457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.043374    6695 scope.go:122] "RemoveContainer" containerID="2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"
Oct 30 21:43:53.043802 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:43:53.043772    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436\": container with ID starting with 2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436 not found: ID does not exist" containerID="2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"
Oct 30 21:43:53.043802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.043826    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436"} err="failed to get container status \"2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436\": rpc error: code = NotFound desc = could not find container \"2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436\": container with ID starting with 2f6828d5d922e45eb24d15babdb6ed54b49d718ab04819ee9a201e30a0ced436 not found: ID does not exist"
Oct 30 21:43:53.078521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.078492    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"]
Oct 30 21:43:53.089077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.089034    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7a4de0-openshift-default-86685696f-g2scm"]
Oct 30 21:43:53.242122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.242021    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" path="/var/lib/kubelet/pods/2c7f4209-3fba-41a2-b098-b81984be14cf/volumes"
Oct 30 21:43:53.242712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:53.242687    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe5d089-eb01-449a-a1fa-c9458937f92e" path="/var/lib/kubelet/pods/2fe5d089-eb01-449a-a1fa-c9458937f92e/volumes"
Oct 30 21:43:58.051901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:58.051862    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"]
Oct 30 21:43:58.075551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:58.075511    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3dedce-n5npc"]
Oct 30 21:43:59.217135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:43:59.217099    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2468efc-f5d8-4900-8d14-7565e567f72f" path="/var/lib/kubelet/pods/c2468efc-f5d8-4900-8d14-7565e567f72f/volumes"
Oct 30 21:44:00.845120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:00.845090    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:44:00.963788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:00.963758    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:44:01.088733 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:44:01.088695    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:44:01.123084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:01.122995    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:44:02.058160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:02.058125    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"]
Oct 30 21:44:02.069312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:02.069272    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea206f8-9q882"]
Oct 30 21:44:03.116945 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.116893    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"]
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148533    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy"
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148565    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy"
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148624    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy"
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148633    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy"
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148859    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy"
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148881    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7f4209-3fba-41a2-b098-b81984be14cf" containerName="istio-proxy"
Oct 30 21:44:03.151173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.148902    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10ff2df-7e2d-4273-b248-250f6f73d33e" containerName="istio-proxy"
Oct 30 21:44:03.158821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.158789    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"]
Oct 30 21:44:03.159754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.159726    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.175863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.175827    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"]
Oct 30 21:44:03.272168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.212249    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a2fc47\""
Oct 30 21:44:03.272450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.213010    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a2fc47-dockercfg-k4l5d\""
Oct 30 21:44:03.272627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.213286    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.357624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.350997    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2w4\" (UniqueName: \"kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4\") pod \"payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") " pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.357624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.351224    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.357624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.351261    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42s8p\" (UniqueName: \"kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p\") pod \"payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.421757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.421563    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") " pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.425829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.422484    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5068dcf-7769-4224-81aa-a1b2189001a8" path="/var/lib/kubelet/pods/b5068dcf-7769-4224-81aa-a1b2189001a8/volumes"
Oct 30 21:44:03.425829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.424463    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"]
Oct 30 21:44:03.524938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.524900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.524938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.524942    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42s8p\" (UniqueName: \"kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p\") pod \"payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.525280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.525253    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") " pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.525385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.525364    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2w4\" (UniqueName: \"kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4\") pod \"payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") " pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.526039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.526015    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") " pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.526148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.526040    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.546098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.546065    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2w4\" (UniqueName: \"kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4\") pod \"payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") " pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:03.546242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.546139    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42s8p\" (UniqueName: \"kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p\") pod \"payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.731738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.731651    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:03.756596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:03.756561    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:04.052292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.052268    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"]
Oct 30 21:44:04.055111 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:44:04.055083    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c92eaaf_658a_423d_8b16_7cfebc93e7b1.slice/crio-407b3a9f6cdb17af3bc2c27fb8d82725a3c8cce68603188168f7f0213ddf1eda WatchSource:0}: Error finding container 407b3a9f6cdb17af3bc2c27fb8d82725a3c8cce68603188168f7f0213ddf1eda: Status 404 returned error can't find the container with id 407b3a9f6cdb17af3bc2c27fb8d82725a3c8cce68603188168f7f0213ddf1eda
Oct 30 21:44:04.097277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.097250    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"]
Oct 30 21:44:04.109098 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:44:04.109059    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d71b7ba_dcc3_4d78_bc95_52b4c0116e58.slice/crio-5f09e860f14b5a00205366c88a79e59e3b869b8226da5ed7dc6130bee9e5f6d7 WatchSource:0}: Error finding container 5f09e860f14b5a00205366c88a79e59e3b869b8226da5ed7dc6130bee9e5f6d7: Status 404 returned error can't find the container with id 5f09e860f14b5a00205366c88a79e59e3b869b8226da5ed7dc6130bee9e5f6d7
Oct 30 21:44:04.168673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.168639    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt" event={"ID":"2c92eaaf-658a-423d-8b16-7cfebc93e7b1","Type":"ContainerStarted","Data":"665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33"}
Oct 30 21:44:04.169046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.168713    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:04.169046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.168739    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt" event={"ID":"2c92eaaf-658a-423d-8b16-7cfebc93e7b1","Type":"ContainerStarted","Data":"407b3a9f6cdb17af3bc2c27fb8d82725a3c8cce68603188168f7f0213ddf1eda"}
Oct 30 21:44:04.170213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.170192    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52" event={"ID":"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58","Type":"ContainerStarted","Data":"5f09e860f14b5a00205366c88a79e59e3b869b8226da5ed7dc6130bee9e5f6d7"}
Oct 30 21:44:04.214387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:04.214334    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt" podStartSLOduration=1.214319505 podStartE2EDuration="1.214319505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:04.20965899 +0000 UTC m=+2027.132117286" watchObservedRunningTime="2026-10-30 21:44:04.214319505 +0000 UTC m=+2027.136777828"
Oct 30 21:44:05.185459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:05.185410    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52" event={"ID":"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58","Type":"ContainerStarted","Data":"41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b"}
Oct 30 21:44:05.185973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:05.185500    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:05.229045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:05.228977    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52" podStartSLOduration=2.228960239 podStartE2EDuration="2.228960239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:05.226786649 +0000 UTC m=+2028.149244947" watchObservedRunningTime="2026-10-30 21:44:05.228960239 +0000 UTC m=+2028.151418538"
Oct 30 21:44:11.028304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:11.028264    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"]
Oct 30 21:44:11.039414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:11.039380    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dbd3d78-886d4"]
Oct 30 21:44:11.216848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:11.216810    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef19ce3b-8bc8-4409-a1b6-33722f8d6da1" path="/var/lib/kubelet/pods/ef19ce3b-8bc8-4409-a1b6-33722f8d6da1/volumes"
Oct 30 21:44:15.028553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:15.028517    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"]
Oct 30 21:44:15.039321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:15.039291    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90915438-6mp2c"]
Oct 30 21:44:15.209825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:15.209793    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:44:15.229150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:15.229115    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8928e6d3-a3b3-4e67-92f7-2f08c16806a3" path="/var/lib/kubelet/pods/8928e6d3-a3b3-4e67-92f7-2f08c16806a3/volumes"
Oct 30 21:44:16.197307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:16.197273    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:44:21.137293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:21.137257    6695 scope.go:122] "RemoveContainer" containerID="bfe556fec3a50346be6aa761910acc426204cf199461640f660e68eb60eba467"
Oct 30 21:44:21.148438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:21.148416    6695 scope.go:122] "RemoveContainer" containerID="87444b7edda468ff757e8a8d816290879309d245729a9f8e9c2413536249c2a4"
Oct 30 21:44:21.158490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:21.158471    6695 scope.go:122] "RemoveContainer" containerID="eda473c0b8f3f8122fccc8b5d6a85a57d7f60dbb02544f10b37a2ba96a0145bd"
Oct 30 21:44:21.170616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:21.170581    6695 scope.go:122] "RemoveContainer" containerID="3d2edbbd328c9427e5bff3ee392e4fea2776120c76b0240ca3f63160ff4649f4"
Oct 30 21:44:21.180931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:21.180910    6695 scope.go:122] "RemoveContainer" containerID="550c8ea09103ec98ce9b72edcc342401b71684faff8ba175c901a8851bee9f46"
Oct 30 21:44:21.190269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:21.190247    6695 scope.go:122] "RemoveContainer" containerID="816dd2bbe875066263c1d1e58e2baa3f349dd99e76913e0f8382f16faeea5977"
Oct 30 21:44:27.030128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:27.030086    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"]
Oct 30 21:44:27.040740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:27.040704    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fe0be9-wncch"]
Oct 30 21:44:27.216781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:27.216750    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca2c8d5-5f36-4c08-9734-ef02b4a2862a" path="/var/lib/kubelet/pods/1ca2c8d5-5f36-4c08-9734-ef02b4a2862a/volumes"
Oct 30 21:44:31.077310 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:44:31.077259    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:44:33.029650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:33.029599    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"]
Oct 30 21:44:33.040196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:33.040162    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f3aaf7-2dsjb"]
Oct 30 21:44:33.215960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:33.215927    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097cfe1d-c121-4908-bf27-be1986f66b69" path="/var/lib/kubelet/pods/097cfe1d-c121-4908-bf27-be1986f66b69/volumes"
Oct 30 21:44:39.029061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:39.029027    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"]
Oct 30 21:44:39.040630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:39.040577    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5a613af-2k7rj"]
Oct 30 21:44:39.224742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:39.224706    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77838753-5eee-48df-9d11-f63b230b6534" path="/var/lib/kubelet/pods/77838753-5eee-48df-9d11-f63b230b6534/volumes"
Oct 30 21:44:49.036984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:49.036950    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"]
Oct 30 21:44:49.048438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:49.048405    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6416a132-m4xzx"]
Oct 30 21:44:49.216904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:49.216867    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06858e92-ee3b-4439-8b97-b602de0bbbb6" path="/var/lib/kubelet/pods/06858e92-ee3b-4439-8b97-b602de0bbbb6/volumes"
Oct 30 21:44:51.026540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:51.026503    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"]
Oct 30 21:44:51.037367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:51.037338    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa13844-gz4cj"]
Oct 30 21:44:51.215215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:51.215177    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fbb120-e52a-4299-b69b-93552ca361a1" path="/var/lib/kubelet/pods/73fbb120-e52a-4299-b69b-93552ca361a1/volumes"
Oct 30 21:44:54.031430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:54.031389    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"]
Oct 30 21:44:54.042118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:54.042093    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa73ef45-5pfsm"]
Oct 30 21:44:55.215261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:44:55.215227    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43921979-2dac-461b-a941-10f700bb797c" path="/var/lib/kubelet/pods/43921979-2dac-461b-a941-10f700bb797c/volumes"
Oct 30 21:45:00.178677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.172653    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"]
Oct 30 21:45:00.209698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.203398    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.235579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.235545    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:45:00.238346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.236388    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"]
Oct 30 21:45:00.241305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.236516    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:45:00.264926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.264893    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"]
Oct 30 21:45:00.267772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.267746    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.327656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.326169    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:45:00.335358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.335319    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"]
Oct 30 21:45:00.335713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.335692    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.340312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.340293    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"]
Oct 30 21:45:00.340865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.340396    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.347173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.347155    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"]
Oct 30 21:45:00.347420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.347263    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.354361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.354048    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"]
Oct 30 21:45:00.354361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.354091    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"]
Oct 30 21:45:00.354361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.354117    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"]
Oct 30 21:45:00.354361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.354142    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"]
Oct 30 21:45:00.354361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.354161    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.361111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.359498    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"]
Oct 30 21:45:00.361111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.359945    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365168    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"]
Oct 30 21:45:00.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365209    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"]
Oct 30 21:45:00.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365264    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365166    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mbg\" (UniqueName: \"kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365623    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfl27\" (UniqueName: \"kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27\") pod \"maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.365814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365788    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.366163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.365822    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.382546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.379837    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"]
Oct 30 21:45:00.388245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.385962    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.419573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.419486    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"]
Oct 30 21:45:00.419573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.419572    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"]
Oct 30 21:45:00.420014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.419639    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"]
Oct 30 21:45:00.420014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.419667    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"]
Oct 30 21:45:00.420014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.419763    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.424886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.424865    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"]
Oct 30 21:45:00.431136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.431117    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"]
Oct 30 21:45:00.467193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467156    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.467361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467229    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.467361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467290    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.467361 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467354    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.467595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467390    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.467595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467501    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2lb\" (UniqueName: \"kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.467595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467528    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kbb\" (UniqueName: \"kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.467769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467678    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mbg\" (UniqueName: \"kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.467769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467726    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.467844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467779    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8xf\" (UniqueName: \"kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.467844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467836    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.467911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467864    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-bshwt\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.467911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467897    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.467987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467925    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw27v\" (UniqueName: \"kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.467987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467954    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l926\" (UniqueName: \"kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.467987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.467977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccr6g\" (UniqueName: \"kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g\") pod \"maas-api-key-cleanup-29889945-bshwt\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.468088 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.468001    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfl27\" (UniqueName: \"kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27\") pod \"maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.468123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.468106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.468183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.468162    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.468278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.468260    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvqw\" (UniqueName: \"kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.490835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.490806    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfl27\" (UniqueName: \"kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27\") pod \"maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.491110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.491088    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mbg\" (UniqueName: \"kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.541836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.541811    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:00.570115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.570081    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.570256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.570175    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.570978    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571008    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.570209    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571056    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571175    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2lb\" (UniqueName: \"kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571280    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kbb\" (UniqueName: \"kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.571392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571332    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571425    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8xf\" (UniqueName: \"kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571525    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571582    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-bshwt\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571713    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571776    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw27v\" (UniqueName: \"kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571839    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l926\" (UniqueName: \"kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571869    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.571939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.571923    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccr6g\" (UniqueName: \"kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g\") pod \"maas-api-key-cleanup-29889945-bshwt\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.572401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.572011    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvqw\" (UniqueName: \"kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.573221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.573193    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.573334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.573198    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.573512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.573485    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.574548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.573788    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-bshwt\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.601691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.601659    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvqw\" (UniqueName: \"kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.604353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.604322    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccr6g\" (UniqueName: \"kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g\") pod \"maas-api-key-cleanup-29889945-bshwt\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.604575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.604551    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2lb\" (UniqueName: \"kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb\") pod \"maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.616075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.613541    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8xf\" (UniqueName: \"kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.616075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.615204    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw27v\" (UniqueName: \"kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.616494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.616467    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kbb\" (UniqueName: \"kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.616726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.616699    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l926\" (UniqueName: \"kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926\") pod \"maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.635663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.635213    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:00.658561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.658535    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:00.677631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.674418    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.677631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.674991    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.677631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.675828    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.679447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.679424    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:00.710559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.710360    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:00.711067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.711043    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:00.715943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.715917    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.732745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.732526    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:00.737893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.737719    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:00.739817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.739765    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:00.752189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.751836    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:00.905885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.889361    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"]
Oct 30 21:45:00.971258 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:00.971113    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfbfcb8_38ac_4ebf_b990_3d482acc76c7.slice/crio-a5eec74dfaf26c67396cd50a896ff196342384296376793e794debf8432da963 WatchSource:0}: Error finding container a5eec74dfaf26c67396cd50a896ff196342384296376793e794debf8432da963: Status 404 returned error can't find the container with id a5eec74dfaf26c67396cd50a896ff196342384296376793e794debf8432da963
Oct 30 21:45:00.973038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:00.973008    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"]
Oct 30 21:45:01.031773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.031738    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"]
Oct 30 21:45:01.040827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.040204    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3061e6b-jqwzt"]
Oct 30 21:45:01.183572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.183532    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"]
Oct 30 21:45:01.215629 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:01.215521    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb149753a_6495_4307_8b00_43ac2de2aeb4.slice/crio-3509393abac5b5a5978d507248c7d1ac39d91fd4e6ed359d64d4a7cfabaf7b98 WatchSource:0}: Error finding container 3509393abac5b5a5978d507248c7d1ac39d91fd4e6ed359d64d4a7cfabaf7b98: Status 404 returned error can't find the container with id 3509393abac5b5a5978d507248c7d1ac39d91fd4e6ed359d64d4a7cfabaf7b98
Oct 30 21:45:01.229556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.229525    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672d4db1-6b2e-400f-94f3-29e24b477885" path="/var/lib/kubelet/pods/672d4db1-6b2e-400f-94f3-29e24b477885/volumes"
Oct 30 21:45:01.230091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.230069    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"]
Oct 30 21:45:01.230951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.230926    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"]
Oct 30 21:45:01.237547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.237520    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"]
Oct 30 21:45:01.237869 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:01.237842    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:45:01.241118 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:01.241095    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c19c73_e32b_40cb_9953_52c0acc2045e.slice/crio-9de9a61e2cc85436fcaeb8a00e155032ab82f6144af62b919bcd7b941af535ea WatchSource:0}: Error finding container 9de9a61e2cc85436fcaeb8a00e155032ab82f6144af62b919bcd7b941af535ea: Status 404 returned error can't find the container with id 9de9a61e2cc85436fcaeb8a00e155032ab82f6144af62b919bcd7b941af535ea
Oct 30 21:45:01.260476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.260452    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"]
Oct 30 21:45:01.268765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.268741    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"]
Oct 30 21:45:01.269193 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:01.269169    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b6452d_d038_404a_808f_5ff935288f7c.slice/crio-8a5527fdb2b462116318f5f13b934edb42446096511f2900e8b0e0e4d5ef3770 WatchSource:0}: Error finding container 8a5527fdb2b462116318f5f13b934edb42446096511f2900e8b0e0e4d5ef3770: Status 404 returned error can't find the container with id 8a5527fdb2b462116318f5f13b934edb42446096511f2900e8b0e0e4d5ef3770
Oct 30 21:45:01.323743 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:01.323708    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c20fd7_4f9f_4c21_a36d_d16830878d44.slice/crio-3d58ba01a7e989fa3c174768fbf1ccc84d6d801496a2be12fa53bde73e7ce9d2 WatchSource:0}: Error finding container 3d58ba01a7e989fa3c174768fbf1ccc84d6d801496a2be12fa53bde73e7ce9d2: Status 404 returned error can't find the container with id 3d58ba01a7e989fa3c174768fbf1ccc84d6d801496a2be12fa53bde73e7ce9d2
Oct 30 21:45:01.328562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.326422    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"]
Oct 30 21:45:01.347317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:01.347292    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"]
Oct 30 21:45:01.416397 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:01.416207    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9795c21d_265d_4220_bf9e_68acade2e7ba.slice/crio-fc464f71d313c7c133508aae74a17dca7779a2b425740b607fd0ec6ba6d49bdb WatchSource:0}: Error finding container fc464f71d313c7c133508aae74a17dca7779a2b425740b607fd0ec6ba6d49bdb: Status 404 returned error can't find the container with id fc464f71d313c7c133508aae74a17dca7779a2b425740b607fd0ec6ba6d49bdb
Oct 30 21:45:01.514946 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:01.514897    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ca70bb_dec2_4342_9cd9_b96be07afe88.slice/crio-1dbf4cee6be16b5149c1b6d73d5f239f26d7891c45056520d0931ae790a92165 WatchSource:0}: Error finding container 1dbf4cee6be16b5149c1b6d73d5f239f26d7891c45056520d0931ae790a92165: Status 404 returned error can't find the container with id 1dbf4cee6be16b5149c1b6d73d5f239f26d7891c45056520d0931ae790a92165
Oct 30 21:45:02.008068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.007999    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" event={"ID":"a5ca70bb-dec2-4342-9cd9-b96be07afe88","Type":"ContainerStarted","Data":"1dbf4cee6be16b5149c1b6d73d5f239f26d7891c45056520d0931ae790a92165"}
Oct 30 21:45:02.015010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.014968    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" event={"ID":"b149753a-6495-4307-8b00-43ac2de2aeb4","Type":"ContainerStarted","Data":"3509393abac5b5a5978d507248c7d1ac39d91fd4e6ed359d64d4a7cfabaf7b98"}
Oct 30 21:45:02.021417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.021384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerStarted","Data":"c425962eb7f8d9a86ad0c9446ce4239392a3f3c801a874d0efcd0c81ef10a1ae"}
Oct 30 21:45:02.021661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.021641    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerStarted","Data":"a5eec74dfaf26c67396cd50a896ff196342384296376793e794debf8432da963"}
Oct 30 21:45:02.028535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.028502    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerStarted","Data":"cf9447e0d13bc13fbbebd8fbd0638a50f5f848d77494a73b3411c9da558ff248"}
Oct 30 21:45:02.028795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.028769    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerStarted","Data":"94249991311c555de4415a3d20f077da5e077e1d929c07bad1539ff154510b15"}
Oct 30 21:45:02.031263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.031242    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" event={"ID":"32b6452d-d038-404a-808f-5ff935288f7c","Type":"ContainerStarted","Data":"8a5527fdb2b462116318f5f13b934edb42446096511f2900e8b0e0e4d5ef3770"}
Oct 30 21:45:02.032860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.032828    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" event={"ID":"52c20fd7-4f9f-4c21-a36d-d16830878d44","Type":"ContainerStarted","Data":"3d58ba01a7e989fa3c174768fbf1ccc84d6d801496a2be12fa53bde73e7ce9d2"}
Oct 30 21:45:02.034356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.034333    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerStarted","Data":"adb84a0eafa2c4b858b027405cddfb22f2b382945a45759e2d760e49c527037f"}
Oct 30 21:45:02.035965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.035943    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerStarted","Data":"3e23eeb691d289a7c087e5bbf2b4343d15f6f5fa40f48077dfcffbf5fd00a589"}
Oct 30 21:45:02.037794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.037771    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" event={"ID":"d4c19c73-e32b-40cb-9953-52c0acc2045e","Type":"ContainerStarted","Data":"9de9a61e2cc85436fcaeb8a00e155032ab82f6144af62b919bcd7b941af535ea"}
Oct 30 21:45:02.039033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.039011    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" event={"ID":"9795c21d-265d-4220-bf9e-68acade2e7ba","Type":"ContainerStarted","Data":"fc464f71d313c7c133508aae74a17dca7779a2b425740b607fd0ec6ba6d49bdb"}
Oct 30 21:45:02.097734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.097567    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" podStartSLOduration=2.097536509 podStartE2EDuration="2.097536509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.05910267 +0000 UTC m=+2084.981560967" watchObservedRunningTime="2026-10-30 21:45:02.097536509 +0000 UTC m=+2085.019994800"
Oct 30 21:45:02.100170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:02.100128    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" podStartSLOduration=2.100116476 podStartE2EDuration="2.100116476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.094866543 +0000 UTC m=+2085.017324842" watchObservedRunningTime="2026-10-30 21:45:02.100116476 +0000 UTC m=+2085.022574775"
Oct 30 21:45:03.055995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.055958    6695 generic.go:356] "Generic (PLEG): container finished" podID="32b6452d-d038-404a-808f-5ff935288f7c" containerID="23fea9b30eb3b67d0436fccd9b93e462f6d72c1921d31ae779db04f77b4808d0" exitCode=60
Oct 30 21:45:03.056458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.056039    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" event={"ID":"32b6452d-d038-404a-808f-5ff935288f7c","Type":"ContainerDied","Data":"23fea9b30eb3b67d0436fccd9b93e462f6d72c1921d31ae779db04f77b4808d0"}
Oct 30 21:45:03.056458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.056419    6695 scope.go:122] "RemoveContainer" containerID="23fea9b30eb3b67d0436fccd9b93e462f6d72c1921d31ae779db04f77b4808d0"
Oct 30 21:45:03.058007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.057985    6695 generic.go:356] "Generic (PLEG): container finished" podID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerID="31c389ceaa9797dbedb2db208bb0be3bb9d58d85c9fbb6fec877f1ff6946ba49" exitCode=60
Oct 30 21:45:03.058125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.058020    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" event={"ID":"52c20fd7-4f9f-4c21-a36d-d16830878d44","Type":"ContainerDied","Data":"31c389ceaa9797dbedb2db208bb0be3bb9d58d85c9fbb6fec877f1ff6946ba49"}
Oct 30 21:45:03.058407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.058326    6695 scope.go:122] "RemoveContainer" containerID="31c389ceaa9797dbedb2db208bb0be3bb9d58d85c9fbb6fec877f1ff6946ba49"
Oct 30 21:45:03.061144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.060286    6695 generic.go:356] "Generic (PLEG): container finished" podID="436d27c5-7a02-4df3-8856-5c540f140536" containerID="b83b4124bd33e87f242138ad4945a93c29b64390d42a66dbe9fff57fd0ff0a6a" exitCode=60
Oct 30 21:45:03.061144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.060387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerDied","Data":"b83b4124bd33e87f242138ad4945a93c29b64390d42a66dbe9fff57fd0ff0a6a"}
Oct 30 21:45:03.061144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.060652    6695 scope.go:122] "RemoveContainer" containerID="b83b4124bd33e87f242138ad4945a93c29b64390d42a66dbe9fff57fd0ff0a6a"
Oct 30 21:45:03.062879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.062849    6695 generic.go:356] "Generic (PLEG): container finished" podID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerID="e1be81cfcf56efd6bb73c2b1fb6d2b139557d2688a1bfb80accf07bdd6d97d85" exitCode=60
Oct 30 21:45:03.062964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.062925    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerDied","Data":"e1be81cfcf56efd6bb73c2b1fb6d2b139557d2688a1bfb80accf07bdd6d97d85"}
Oct 30 21:45:03.063142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.063119    6695 scope.go:122] "RemoveContainer" containerID="e1be81cfcf56efd6bb73c2b1fb6d2b139557d2688a1bfb80accf07bdd6d97d85"
Oct 30 21:45:03.064682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.064661    6695 generic.go:356] "Generic (PLEG): container finished" podID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerID="c9e1b8548ec14551e6b260efeef2e72395ba4646bfc37b78321ee8f1ada29599" exitCode=60
Oct 30 21:45:03.064812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.064792    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" event={"ID":"d4c19c73-e32b-40cb-9953-52c0acc2045e","Type":"ContainerDied","Data":"c9e1b8548ec14551e6b260efeef2e72395ba4646bfc37b78321ee8f1ada29599"}
Oct 30 21:45:03.064974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.064960    6695 scope.go:122] "RemoveContainer" containerID="c9e1b8548ec14551e6b260efeef2e72395ba4646bfc37b78321ee8f1ada29599"
Oct 30 21:45:03.066362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.066340    6695 generic.go:356] "Generic (PLEG): container finished" podID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerID="c6cea3f38d1b953a435243d0471fe63303b9f1d5ce83e8f742108a6e792412e3" exitCode=60
Oct 30 21:45:03.066476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.066419    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" event={"ID":"9795c21d-265d-4220-bf9e-68acade2e7ba","Type":"ContainerDied","Data":"c6cea3f38d1b953a435243d0471fe63303b9f1d5ce83e8f742108a6e792412e3"}
Oct 30 21:45:03.066579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.066563    6695 scope.go:122] "RemoveContainer" containerID="c6cea3f38d1b953a435243d0471fe63303b9f1d5ce83e8f742108a6e792412e3"
Oct 30 21:45:03.068149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.068125    6695 generic.go:356] "Generic (PLEG): container finished" podID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerID="cf829a5d2c51118fed4b9cbc4fe00d0a9a58d5cb60e24fb29d10ba2dd203d6b5" exitCode=60
Oct 30 21:45:03.068250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.068195    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" event={"ID":"a5ca70bb-dec2-4342-9cd9-b96be07afe88","Type":"ContainerDied","Data":"cf829a5d2c51118fed4b9cbc4fe00d0a9a58d5cb60e24fb29d10ba2dd203d6b5"}
Oct 30 21:45:03.068421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.068405    6695 scope.go:122] "RemoveContainer" containerID="cf829a5d2c51118fed4b9cbc4fe00d0a9a58d5cb60e24fb29d10ba2dd203d6b5"
Oct 30 21:45:03.069846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.069826    6695 generic.go:356] "Generic (PLEG): container finished" podID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerID="c3184f56b5b770c8b411e6c8228ca3d1bc8b41e33011acf10b137f88fd8c6efc" exitCode=60
Oct 30 21:45:03.069948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.069909    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" event={"ID":"b149753a-6495-4307-8b00-43ac2de2aeb4","Type":"ContainerDied","Data":"c3184f56b5b770c8b411e6c8228ca3d1bc8b41e33011acf10b137f88fd8c6efc"}
Oct 30 21:45:03.070137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.070121    6695 scope.go:122] "RemoveContainer" containerID="c3184f56b5b770c8b411e6c8228ca3d1bc8b41e33011acf10b137f88fd8c6efc"
Oct 30 21:45:03.073693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.073670    6695 generic.go:356] "Generic (PLEG): container finished" podID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerID="c425962eb7f8d9a86ad0c9446ce4239392a3f3c801a874d0efcd0c81ef10a1ae" exitCode=60
Oct 30 21:45:03.073787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.073744    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerDied","Data":"c425962eb7f8d9a86ad0c9446ce4239392a3f3c801a874d0efcd0c81ef10a1ae"}
Oct 30 21:45:03.073960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.073945    6695 scope.go:122] "RemoveContainer" containerID="c425962eb7f8d9a86ad0c9446ce4239392a3f3c801a874d0efcd0c81ef10a1ae"
Oct 30 21:45:03.075330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.075309    6695 generic.go:356] "Generic (PLEG): container finished" podID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerID="cf9447e0d13bc13fbbebd8fbd0638a50f5f848d77494a73b3411c9da558ff248" exitCode=60
Oct 30 21:45:03.075410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.075343    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerDied","Data":"cf9447e0d13bc13fbbebd8fbd0638a50f5f848d77494a73b3411c9da558ff248"}
Oct 30 21:45:03.075617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:03.075586    6695 scope.go:122] "RemoveContainer" containerID="cf9447e0d13bc13fbbebd8fbd0638a50f5f848d77494a73b3411c9da558ff248"
Oct 30 21:45:04.030784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.030707    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"]
Oct 30 21:45:04.035017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.034994    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cd644d0-z9h4g"]
Oct 30 21:45:04.039078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.038971    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"]
Oct 30 21:45:04.044042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.044010    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc68434d-49bw8"]
Oct 30 21:45:04.087441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.087405    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerStarted","Data":"0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943"}
Oct 30 21:45:04.089665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.089596    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerStarted","Data":"be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c"}
Oct 30 21:45:04.091548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.091522    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" event={"ID":"32b6452d-d038-404a-808f-5ff935288f7c","Type":"ContainerStarted","Data":"2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1"}
Oct 30 21:45:04.093793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.093766    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" event={"ID":"52c20fd7-4f9f-4c21-a36d-d16830878d44","Type":"ContainerStarted","Data":"152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5"}
Oct 30 21:45:04.095850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.095824    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerStarted","Data":"585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b"}
Oct 30 21:45:04.097617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.097584    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerStarted","Data":"edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab"}
Oct 30 21:45:04.099470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.099446    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" event={"ID":"d4c19c73-e32b-40cb-9953-52c0acc2045e","Type":"ContainerStarted","Data":"b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f"}
Oct 30 21:45:04.101561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.101535    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" event={"ID":"b149753a-6495-4307-8b00-43ac2de2aeb4","Type":"ContainerStarted","Data":"546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112"}
Oct 30 21:45:04.166929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.166876    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" podStartSLOduration=4.166862494 podStartE2EDuration="4.166862494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.165746589 +0000 UTC m=+2087.088204887" watchObservedRunningTime="2026-10-30 21:45:04.166862494 +0000 UTC m=+2087.089320791"
Oct 30 21:45:04.203479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.203419    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" podStartSLOduration=4.203404441 podStartE2EDuration="4.203404441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.201667558 +0000 UTC m=+2087.124125858" watchObservedRunningTime="2026-10-30 21:45:04.203404441 +0000 UTC m=+2087.125862731"
Oct 30 21:45:04.239178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.239129    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" podStartSLOduration=4.239114345 podStartE2EDuration="4.239114345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.238290786 +0000 UTC m=+2087.160749084" watchObservedRunningTime="2026-10-30 21:45:04.239114345 +0000 UTC m=+2087.161572643"
Oct 30 21:45:04.276034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.275927    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" podStartSLOduration=4.275914782 podStartE2EDuration="4.275914782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.274744134 +0000 UTC m=+2087.197202447" watchObservedRunningTime="2026-10-30 21:45:04.275914782 +0000 UTC m=+2087.198373080"
Oct 30 21:45:04.320517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.320457    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" podStartSLOduration=4.320437154 podStartE2EDuration="4.320437154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.315939954 +0000 UTC m=+2087.238398263" watchObservedRunningTime="2026-10-30 21:45:04.320437154 +0000 UTC m=+2087.242895453"
Oct 30 21:45:04.389624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:04.389545    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" podStartSLOduration=4.389524807 podStartE2EDuration="4.389524807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.387318096 +0000 UTC m=+2087.309776395" watchObservedRunningTime="2026-10-30 21:45:04.389524807 +0000 UTC m=+2087.311983105"
Oct 30 21:45:05.038836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.038800    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"]
Oct 30 21:45:05.049069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.049034    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e57cc4-ck9zh"]
Oct 30 21:45:05.161942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.158406    6695 scope.go:122] "RemoveContainer" containerID="2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1"
Oct 30 21:45:05.161942 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.158987    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-bshwt_odh-ai-gateway-infra(32b6452d-d038-404a-808f-5ff935288f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" podUID="32b6452d-d038-404a-808f-5ff935288f7c"
Oct 30 21:45:05.161942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.159537    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" event={"ID":"32b6452d-d038-404a-808f-5ff935288f7c","Type":"ContainerDied","Data":"2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1"}
Oct 30 21:45:05.161942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.159699    6695 scope.go:122] "RemoveContainer" containerID="23fea9b30eb3b67d0436fccd9b93e462f6d72c1921d31ae779db04f77b4808d0"
Oct 30 21:45:05.161942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.157687    6695 generic.go:356] "Generic (PLEG): container finished" podID="32b6452d-d038-404a-808f-5ff935288f7c" containerID="2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1" exitCode=60
Oct 30 21:45:05.194435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.193285    6695 generic.go:356] "Generic (PLEG): container finished" podID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerID="152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5" exitCode=60
Oct 30 21:45:05.194435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.193405    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" event={"ID":"52c20fd7-4f9f-4c21-a36d-d16830878d44","Type":"ContainerDied","Data":"152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5"}
Oct 30 21:45:05.194435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.193891    6695 scope.go:122] "RemoveContainer" containerID="152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5"
Oct 30 21:45:05.194435 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.194335    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh_odh-ai-gateway-infra(52c20fd7-4f9f-4c21-a36d-d16830878d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44"
Oct 30 21:45:05.199211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.199185    6695 generic.go:356] "Generic (PLEG): container finished" podID="436d27c5-7a02-4df3-8856-5c540f140536" containerID="585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b" exitCode=60
Oct 30 21:45:05.199355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.199257    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerDied","Data":"585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b"}
Oct 30 21:45:05.199646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.199622    6695 scope.go:122] "RemoveContainer" containerID="585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b"
Oct 30 21:45:05.200010 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.199987    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6_odh-ai-gateway-infra(436d27c5-7a02-4df3-8856-5c540f140536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" podUID="436d27c5-7a02-4df3-8856-5c540f140536"
Oct 30 21:45:05.201364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.201346    6695 scope.go:122] "RemoveContainer" containerID="31c389ceaa9797dbedb2db208bb0be3bb9d58d85c9fbb6fec877f1ff6946ba49"
Oct 30 21:45:05.201818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.201798    6695 generic.go:356] "Generic (PLEG): container finished" podID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerID="edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab" exitCode=60
Oct 30 21:45:05.201962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.201860    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerDied","Data":"edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab"}
Oct 30 21:45:05.202318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.202250    6695 scope.go:122] "RemoveContainer" containerID="edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab"
Oct 30 21:45:05.202730 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.202705    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8_odh-ai-gateway-infra(f8fc6bcc-9a99-44bd-85c0-802f7320912a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a"
Oct 30 21:45:05.205053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.205030    6695 generic.go:356] "Generic (PLEG): container finished" podID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerID="b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f" exitCode=60
Oct 30 21:45:05.205194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.205176    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" event={"ID":"d4c19c73-e32b-40cb-9953-52c0acc2045e","Type":"ContainerDied","Data":"b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f"}
Oct 30 21:45:05.205395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.205380    6695 scope.go:122] "RemoveContainer" containerID="b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f"
Oct 30 21:45:05.205701 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.205681    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb_odh-ai-gateway-infra(d4c19c73-e32b-40cb-9953-52c0acc2045e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e"
Oct 30 21:45:05.207803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.207781    6695 generic.go:356] "Generic (PLEG): container finished" podID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerID="26d8eb409b6602173730a22fa8e9a7099084f8bb4380d7b1d8aad26d6c9f33da" exitCode=60
Oct 30 21:45:05.208308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.208139    6695 scope.go:122] "RemoveContainer" containerID="26d8eb409b6602173730a22fa8e9a7099084f8bb4380d7b1d8aad26d6c9f33da"
Oct 30 21:45:05.208429 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.208400    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2_odh-ai-gateway-infra(9795c21d-265d-4220-bf9e-68acade2e7ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba"
Oct 30 21:45:05.210059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.210039    6695 generic.go:356] "Generic (PLEG): container finished" podID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerID="7ffb5c443de915109e00c09b22bee80fc1b939576b6c169ee2b8d819fc69aa95" exitCode=60
Oct 30 21:45:05.210328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.210313    6695 scope.go:122] "RemoveContainer" containerID="7ffb5c443de915109e00c09b22bee80fc1b939576b6c169ee2b8d819fc69aa95"
Oct 30 21:45:05.210505 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.210488    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt_odh-ai-gateway-infra(a5ca70bb-dec2-4342-9cd9-b96be07afe88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88"
Oct 30 21:45:05.212416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.212391    6695 generic.go:356] "Generic (PLEG): container finished" podID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerID="546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112" exitCode=60
Oct 30 21:45:05.212416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.212811    6695 scope.go:122] "RemoveContainer" containerID="546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112"
Oct 30 21:45:05.212416 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.213046    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl_odh-ai-gateway-infra(b149753a-6495-4307-8b00-43ac2de2aeb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4"
Oct 30 21:45:05.215345 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.215313    6695 generic.go:356] "Generic (PLEG): container finished" podID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerID="0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943" exitCode=60
Oct 30 21:45:05.215687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.215668    6695 scope.go:122] "RemoveContainer" containerID="0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943"
Oct 30 21:45:05.215923 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.215899    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv_odh-ai-gateway-infra(4cfbfcb8-38ac-4ebf-b990-3d482acc76c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7"
Oct 30 21:45:05.217395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.217375    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab98276-242e-4bae-a6dc-4114927e5930" path="/var/lib/kubelet/pods/3ab98276-242e-4bae-a6dc-4114927e5930/volumes"
Oct 30 21:45:05.217720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.217699    6695 generic.go:356] "Generic (PLEG): container finished" podID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerID="be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c" exitCode=60
Oct 30 21:45:05.217954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.217936    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b721028d-8260-4a62-90f4-06d45e308405" path="/var/lib/kubelet/pods/b721028d-8260-4a62-90f4-06d45e308405/volumes"
Oct 30 21:45:05.218042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.218026    6695 scope.go:122] "RemoveContainer" containerID="be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c"
Oct 30 21:45:05.218303 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:05.218284    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls_odh-ai-gateway-infra(c3c10d60-1531-4f58-a2f9-a3b129a7c4f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4"
Oct 30 21:45:05.218453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.218411    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee" path="/var/lib/kubelet/pods/badfcda4-a93d-4c3e-9e1e-d6d1a866c7ee/volumes"
Oct 30 21:45:05.219238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.219217    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" event={"ID":"9795c21d-265d-4220-bf9e-68acade2e7ba","Type":"ContainerDied","Data":"26d8eb409b6602173730a22fa8e9a7099084f8bb4380d7b1d8aad26d6c9f33da"}
Oct 30 21:45:05.219325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.219266    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" event={"ID":"a5ca70bb-dec2-4342-9cd9-b96be07afe88","Type":"ContainerDied","Data":"7ffb5c443de915109e00c09b22bee80fc1b939576b6c169ee2b8d819fc69aa95"}
Oct 30 21:45:05.219325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.219301    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" event={"ID":"b149753a-6495-4307-8b00-43ac2de2aeb4","Type":"ContainerDied","Data":"546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112"}
Oct 30 21:45:05.219325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.219320    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerDied","Data":"0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943"}
Oct 30 21:45:05.219453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.219337    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerDied","Data":"be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c"}
Oct 30 21:45:05.220050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.220030    6695 scope.go:122] "RemoveContainer" containerID="b83b4124bd33e87f242138ad4945a93c29b64390d42a66dbe9fff57fd0ff0a6a"
Oct 30 21:45:05.234460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.234443    6695 scope.go:122] "RemoveContainer" containerID="e1be81cfcf56efd6bb73c2b1fb6d2b139557d2688a1bfb80accf07bdd6d97d85"
Oct 30 21:45:05.249422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.249401    6695 scope.go:122] "RemoveContainer" containerID="c9e1b8548ec14551e6b260efeef2e72395ba4646bfc37b78321ee8f1ada29599"
Oct 30 21:45:05.264507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.264486    6695 scope.go:122] "RemoveContainer" containerID="c6cea3f38d1b953a435243d0471fe63303b9f1d5ce83e8f742108a6e792412e3"
Oct 30 21:45:05.278878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.278854    6695 scope.go:122] "RemoveContainer" containerID="cf829a5d2c51118fed4b9cbc4fe00d0a9a58d5cb60e24fb29d10ba2dd203d6b5"
Oct 30 21:45:05.293975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.293955    6695 scope.go:122] "RemoveContainer" containerID="c3184f56b5b770c8b411e6c8228ca3d1bc8b41e33011acf10b137f88fd8c6efc"
Oct 30 21:45:05.308815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.308793    6695 scope.go:122] "RemoveContainer" containerID="c425962eb7f8d9a86ad0c9446ce4239392a3f3c801a874d0efcd0c81ef10a1ae"
Oct 30 21:45:05.328670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:05.328647    6695 scope.go:122] "RemoveContainer" containerID="cf9447e0d13bc13fbbebd8fbd0638a50f5f848d77494a73b3411c9da558ff248"
Oct 30 21:45:06.228424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.228395    6695 scope.go:122] "RemoveContainer" containerID="0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943"
Oct 30 21:45:06.228900 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.228804    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv_odh-ai-gateway-infra(4cfbfcb8-38ac-4ebf-b990-3d482acc76c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7"
Oct 30 21:45:06.230123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.230101    6695 scope.go:122] "RemoveContainer" containerID="be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c"
Oct 30 21:45:06.230455 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.230436    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls_odh-ai-gateway-infra(c3c10d60-1531-4f58-a2f9-a3b129a7c4f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4"
Oct 30 21:45:06.231738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.231719    6695 scope.go:122] "RemoveContainer" containerID="2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1"
Oct 30 21:45:06.232010 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.231988    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-bshwt_odh-ai-gateway-infra(32b6452d-d038-404a-808f-5ff935288f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" podUID="32b6452d-d038-404a-808f-5ff935288f7c"
Oct 30 21:45:06.233193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.233172    6695 scope.go:122] "RemoveContainer" containerID="152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5"
Oct 30 21:45:06.233466 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.233445    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh_odh-ai-gateway-infra(52c20fd7-4f9f-4c21-a36d-d16830878d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44"
Oct 30 21:45:06.234740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.234722    6695 scope.go:122] "RemoveContainer" containerID="585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b"
Oct 30 21:45:06.235040 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.235013    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6_odh-ai-gateway-infra(436d27c5-7a02-4df3-8856-5c540f140536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" podUID="436d27c5-7a02-4df3-8856-5c540f140536"
Oct 30 21:45:06.236251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.236228    6695 scope.go:122] "RemoveContainer" containerID="edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab"
Oct 30 21:45:06.236517 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.236496    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8_odh-ai-gateway-infra(f8fc6bcc-9a99-44bd-85c0-802f7320912a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a"
Oct 30 21:45:06.237842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.237825    6695 scope.go:122] "RemoveContainer" containerID="b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f"
Oct 30 21:45:06.238076 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.238046    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb_odh-ai-gateway-infra(d4c19c73-e32b-40cb-9953-52c0acc2045e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e"
Oct 30 21:45:06.239332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.239314    6695 scope.go:122] "RemoveContainer" containerID="26d8eb409b6602173730a22fa8e9a7099084f8bb4380d7b1d8aad26d6c9f33da"
Oct 30 21:45:06.239532 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.239508    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2_odh-ai-gateway-infra(9795c21d-265d-4220-bf9e-68acade2e7ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba"
Oct 30 21:45:06.240853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.240829    6695 scope.go:122] "RemoveContainer" containerID="7ffb5c443de915109e00c09b22bee80fc1b939576b6c169ee2b8d819fc69aa95"
Oct 30 21:45:06.241053 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.241033    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt_odh-ai-gateway-infra(a5ca70bb-dec2-4342-9cd9-b96be07afe88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88"
Oct 30 21:45:06.242449 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:06.242433    6695 scope.go:122] "RemoveContainer" containerID="546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112"
Oct 30 21:45:06.242670 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:06.242651    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl_odh-ai-gateway-infra(b149753a-6495-4307-8b00-43ac2de2aeb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4"
Oct 30 21:45:07.348523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.348482    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"]
Oct 30 21:45:07.355452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.355431    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.376117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.376022    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"]
Oct 30 21:45:07.382397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.382010    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:45:07.525916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.525881    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.526102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.525942    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.526163 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.526134    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrgl\" (UniqueName: \"kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.627641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.627520    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.627641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.627569    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.627902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.627693    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrgl\" (UniqueName: \"kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.628431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.628404    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.630069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.630047    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.648432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.648387    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrgl\" (UniqueName: \"kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl\") pod \"maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.671313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.671284    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:07.978546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:07.978520    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"]
Oct 30 21:45:07.980978 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:07.980942    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d1e4dd_76e1_4c25_b446_4232be74a130.slice/crio-4f215201b74efc2f4947a8b47895f04581422911a936e58480732ac4da646903 WatchSource:0}: Error finding container 4f215201b74efc2f4947a8b47895f04581422911a936e58480732ac4da646903: Status 404 returned error can't find the container with id 4f215201b74efc2f4947a8b47895f04581422911a936e58480732ac4da646903
Oct 30 21:45:08.262087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:08.262045    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9" event={"ID":"c0d1e4dd-76e1-4c25-b446-4232be74a130","Type":"ContainerStarted","Data":"4f215201b74efc2f4947a8b47895f04581422911a936e58480732ac4da646903"}
Oct 30 21:45:09.273016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:09.272977    6695 generic.go:356] "Generic (PLEG): container finished" podID="c0d1e4dd-76e1-4c25-b446-4232be74a130" containerID="82a8deeb6182cd5d1c5447686d646f07a08aeb81124d76c1adb5649737649216" exitCode=0
Oct 30 21:45:09.273424 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:09.273064    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9" event={"ID":"c0d1e4dd-76e1-4c25-b446-4232be74a130","Type":"ContainerDied","Data":"82a8deeb6182cd5d1c5447686d646f07a08aeb81124d76c1adb5649737649216"}
Oct 30 21:45:10.440498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.440474    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:10.475524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.475494    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca\") pod \"c0d1e4dd-76e1-4c25-b446-4232be74a130\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") "
Oct 30 21:45:10.475771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.475755    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca\") pod \"c0d1e4dd-76e1-4c25-b446-4232be74a130\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") "
Oct 30 21:45:10.475854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.475842    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl\" (UniqueName: \"kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl\") pod \"c0d1e4dd-76e1-4c25-b446-4232be74a130\" (UID: \"c0d1e4dd-76e1-4c25-b446-4232be74a130\") "
Oct 30 21:45:10.476120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.476097    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca" pod "c0d1e4dd-76e1-4c25-b446-4232be74a130" (UID: "c0d1e4dd-76e1-4c25-b446-4232be74a130"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:10.476438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.476420    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:10.477551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.477530    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca" pod "c0d1e4dd-76e1-4c25-b446-4232be74a130" (UID: "c0d1e4dd-76e1-4c25-b446-4232be74a130"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:10.477928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.477903    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl" pod "c0d1e4dd-76e1-4c25-b446-4232be74a130" (UID: "c0d1e4dd-76e1-4c25-b446-4232be74a130"). InnerVolumeSpecName "kube-api-access-gmrgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:10.577557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.577476    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmrgl\" (UniqueName: \"kubernetes.io/projected/c0d1e4dd-76e1-4c25-b446-4232be74a130-kube-api-access-gmrgl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:10.577557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:10.577509    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1e4dd-76e1-4c25-b446-4232be74a130-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:11.290980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:11.290951    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"
Oct 30 21:45:11.290980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:11.290975    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9" event={"ID":"c0d1e4dd-76e1-4c25-b446-4232be74a130","Type":"ContainerDied","Data":"4f215201b74efc2f4947a8b47895f04581422911a936e58480732ac4da646903"}
Oct 30 21:45:11.291196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:11.291022    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f215201b74efc2f4947a8b47895f04581422911a936e58480732ac4da646903"
Oct 30 21:45:15.207121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:15.207089    6695 scope.go:122] "RemoveContainer" containerID="7ffb5c443de915109e00c09b22bee80fc1b939576b6c169ee2b8d819fc69aa95"
Oct 30 21:45:15.207511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:15.207213    6695 scope.go:122] "RemoveContainer" containerID="546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112"
Oct 30 21:45:15.207511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:15.207266    6695 scope.go:122] "RemoveContainer" containerID="152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5"
Oct 30 21:45:16.207005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.206977    6695 scope.go:122] "RemoveContainer" containerID="be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c"
Oct 30 21:45:16.207378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.207354    6695 scope.go:122] "RemoveContainer" containerID="0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943"
Oct 30 21:45:16.207781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.207411    6695 scope.go:122] "RemoveContainer" containerID="585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b"
Oct 30 21:45:16.334749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.334719    6695 generic.go:356] "Generic (PLEG): container finished" podID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerID="c42d8277a786213e13113b260f7d0a95880af9b6bdc01b90e3e0f0a81eaf0159" exitCode=60
Oct 30 21:45:16.334892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.334794    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" event={"ID":"a5ca70bb-dec2-4342-9cd9-b96be07afe88","Type":"ContainerDied","Data":"c42d8277a786213e13113b260f7d0a95880af9b6bdc01b90e3e0f0a81eaf0159"}
Oct 30 21:45:16.334892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.334855    6695 scope.go:122] "RemoveContainer" containerID="7ffb5c443de915109e00c09b22bee80fc1b939576b6c169ee2b8d819fc69aa95"
Oct 30 21:45:16.335336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.335314    6695 scope.go:122] "RemoveContainer" containerID="c42d8277a786213e13113b260f7d0a95880af9b6bdc01b90e3e0f0a81eaf0159"
Oct 30 21:45:16.335766 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:16.335735    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt_odh-ai-gateway-infra(a5ca70bb-dec2-4342-9cd9-b96be07afe88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88"
Oct 30 21:45:16.337150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.337123    6695 generic.go:356] "Generic (PLEG): container finished" podID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerID="7b0fa4688c71bf97b5dd214b704c33c90f62d75c4b43d94cbdd5a650ffa75515" exitCode=60
Oct 30 21:45:16.337244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.337193    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" event={"ID":"b149753a-6495-4307-8b00-43ac2de2aeb4","Type":"ContainerDied","Data":"7b0fa4688c71bf97b5dd214b704c33c90f62d75c4b43d94cbdd5a650ffa75515"}
Oct 30 21:45:16.337701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.337674    6695 scope.go:122] "RemoveContainer" containerID="7b0fa4688c71bf97b5dd214b704c33c90f62d75c4b43d94cbdd5a650ffa75515"
Oct 30 21:45:16.338122 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:16.338096    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl_odh-ai-gateway-infra(b149753a-6495-4307-8b00-43ac2de2aeb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4"
Oct 30 21:45:16.339355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.339334    6695 generic.go:356] "Generic (PLEG): container finished" podID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerID="50d5d5a36ecb031e08f0e14cd1199d5407fc4bb6dd510e438eb63bdfd7ad5550" exitCode=60
Oct 30 21:45:16.339426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.339370    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" event={"ID":"52c20fd7-4f9f-4c21-a36d-d16830878d44","Type":"ContainerDied","Data":"50d5d5a36ecb031e08f0e14cd1199d5407fc4bb6dd510e438eb63bdfd7ad5550"}
Oct 30 21:45:16.339682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.339668    6695 scope.go:122] "RemoveContainer" containerID="50d5d5a36ecb031e08f0e14cd1199d5407fc4bb6dd510e438eb63bdfd7ad5550"
Oct 30 21:45:16.339960 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:16.339945    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh_odh-ai-gateway-infra(52c20fd7-4f9f-4c21-a36d-d16830878d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44"
Oct 30 21:45:16.366759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.366740    6695 scope.go:122] "RemoveContainer" containerID="546ab16051a097ac09e85d21c58a038698d9c91c284aae34cfd48935f1535112"
Oct 30 21:45:16.388761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:16.388723    6695 scope.go:122] "RemoveContainer" containerID="152d586c581c01c65b2f4f9145568a79cc926ecceca4b8df9d042e01982b63b5"
Oct 30 21:45:17.223107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.223020    6695 scope.go:122] "RemoveContainer" containerID="edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab"
Oct 30 21:45:17.352934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.352904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerStarted","Data":"735d973386c95cd721deffdd62c25565ae112334e547fb2449386f19b2e61393"}
Oct 30 21:45:17.358015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.357989    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerStarted","Data":"51fb43fb6d45ef50309328839f4c48f914c25364960149cccac5954ed24943df"}
Oct 30 21:45:17.359891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.359864    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerStarted","Data":"6ffd8a07bf5c762a05484cdabc3429cb5c9c0d1984d3b9b5e1690236c3bec92f"}
Oct 30 21:45:17.386156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.386126    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"]
Oct 30 21:45:17.427069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.427035    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"]
Oct 30 21:45:17.479262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.464862    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"]
Oct 30 21:45:17.712396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.712367    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:17.773974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.773896    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:17.791727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.791698    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:17.803860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.803837    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca\") pod \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") "
Oct 30 21:45:17.804031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.804017    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb\" (UniqueName: \"kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb\") pod \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\" (UID: \"a5ca70bb-dec2-4342-9cd9-b96be07afe88\") "
Oct 30 21:45:17.804183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.804161    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca" pod "a5ca70bb-dec2-4342-9cd9-b96be07afe88" (UID: "a5ca70bb-dec2-4342-9cd9-b96be07afe88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:17.804776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.804759    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca70bb-dec2-4342-9cd9-b96be07afe88-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:17.806266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.806242    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb" pod "a5ca70bb-dec2-4342-9cd9-b96be07afe88" (UID: "a5ca70bb-dec2-4342-9cd9-b96be07afe88"). InnerVolumeSpecName "kube-api-access-p7kbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:17.845667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.845630    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"]
Oct 30 21:45:17.883462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.883428    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"]
Oct 30 21:45:17.906128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.906024    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca\") pod \"b149753a-6495-4307-8b00-43ac2de2aeb4\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") "
Oct 30 21:45:17.906440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.906277    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v\" (UniqueName: \"kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v\") pod \"b149753a-6495-4307-8b00-43ac2de2aeb4\" (UID: \"b149753a-6495-4307-8b00-43ac2de2aeb4\") "
Oct 30 21:45:17.906590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.906326    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca\") pod \"52c20fd7-4f9f-4c21-a36d-d16830878d44\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") "
Oct 30 21:45:17.906707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.906686    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926\" (UniqueName: \"kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926\") pod \"52c20fd7-4f9f-4c21-a36d-d16830878d44\" (UID: \"52c20fd7-4f9f-4c21-a36d-d16830878d44\") "
Oct 30 21:45:17.907167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.907140    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca" pod "b149753a-6495-4307-8b00-43ac2de2aeb4" (UID: "b149753a-6495-4307-8b00-43ac2de2aeb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:17.907167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.907140    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca" pod "52c20fd7-4f9f-4c21-a36d-d16830878d44" (UID: "52c20fd7-4f9f-4c21-a36d-d16830878d44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:17.907518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.907497    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b149753a-6495-4307-8b00-43ac2de2aeb4-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:17.907656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.907524    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kbb\" (UniqueName: \"kubernetes.io/projected/a5ca70bb-dec2-4342-9cd9-b96be07afe88-kube-api-access-p7kbb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:17.907656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.907535    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c20fd7-4f9f-4c21-a36d-d16830878d44-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:17.909196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.909169    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v" pod "b149753a-6495-4307-8b00-43ac2de2aeb4" (UID: "b149753a-6495-4307-8b00-43ac2de2aeb4"). InnerVolumeSpecName "kube-api-access-vw27v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:17.909288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:17.909202    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926" pod "52c20fd7-4f9f-4c21-a36d-d16830878d44" (UID: "52c20fd7-4f9f-4c21-a36d-d16830878d44"). InnerVolumeSpecName "kube-api-access-5l926". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:18.009311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.009215    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw27v\" (UniqueName: \"kubernetes.io/projected/b149753a-6495-4307-8b00-43ac2de2aeb4-kube-api-access-vw27v\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:18.009311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.009246    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l926\" (UniqueName: \"kubernetes.io/projected/52c20fd7-4f9f-4c21-a36d-d16830878d44-kube-api-access-5l926\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:18.206527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.206490    6695 scope.go:122] "RemoveContainer" containerID="2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1"
Oct 30 21:45:18.370944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.370909    6695 generic.go:356] "Generic (PLEG): container finished" podID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerID="51fb43fb6d45ef50309328839f4c48f914c25364960149cccac5954ed24943df" exitCode=60
Oct 30 21:45:18.371360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.370978    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerDied","Data":"51fb43fb6d45ef50309328839f4c48f914c25364960149cccac5954ed24943df"}
Oct 30 21:45:18.371360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.371030    6695 scope.go:122] "RemoveContainer" containerID="0b3820fe9f41f0bca509ee6c56e0888ef3c4dd9ea8ccbf5fb5453915b2826943"
Oct 30 21:45:18.371486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.371419    6695 scope.go:122] "RemoveContainer" containerID="51fb43fb6d45ef50309328839f4c48f914c25364960149cccac5954ed24943df"
Oct 30 21:45:18.371804 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:18.371784    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv_odh-ai-gateway-infra(4cfbfcb8-38ac-4ebf-b990-3d482acc76c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7"
Oct 30 21:45:18.373257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.373238    6695 generic.go:356] "Generic (PLEG): container finished" podID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerID="6ffd8a07bf5c762a05484cdabc3429cb5c9c0d1984d3b9b5e1690236c3bec92f" exitCode=60
Oct 30 21:45:18.373344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.373267    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerDied","Data":"6ffd8a07bf5c762a05484cdabc3429cb5c9c0d1984d3b9b5e1690236c3bec92f"}
Oct 30 21:45:18.373696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.373672    6695 scope.go:122] "RemoveContainer" containerID="6ffd8a07bf5c762a05484cdabc3429cb5c9c0d1984d3b9b5e1690236c3bec92f"
Oct 30 21:45:18.374271 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:18.373987    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls_odh-ai-gateway-infra(c3c10d60-1531-4f58-a2f9-a3b129a7c4f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4"
Oct 30 21:45:18.374685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.374667    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"
Oct 30 21:45:18.374761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.374685    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh" event={"ID":"52c20fd7-4f9f-4c21-a36d-d16830878d44","Type":"ContainerDied","Data":"3d58ba01a7e989fa3c174768fbf1ccc84d6d801496a2be12fa53bde73e7ce9d2"}
Oct 30 21:45:18.376714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.376693    6695 generic.go:356] "Generic (PLEG): container finished" podID="436d27c5-7a02-4df3-8856-5c540f140536" containerID="735d973386c95cd721deffdd62c25565ae112334e547fb2449386f19b2e61393" exitCode=60
Oct 30 21:45:18.376812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.376775    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerDied","Data":"735d973386c95cd721deffdd62c25565ae112334e547fb2449386f19b2e61393"}
Oct 30 21:45:18.377180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.377163    6695 scope.go:122] "RemoveContainer" containerID="735d973386c95cd721deffdd62c25565ae112334e547fb2449386f19b2e61393"
Oct 30 21:45:18.377450 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:18.377428    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6_odh-ai-gateway-infra(436d27c5-7a02-4df3-8856-5c540f140536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" podUID="436d27c5-7a02-4df3-8856-5c540f140536"
Oct 30 21:45:18.378946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.378919    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerStarted","Data":"ca91c5195adb0f0030cce8dfa43a65ecc50b3352bd8bda8322a67c0e8354bae2"}
Oct 30 21:45:18.381866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.381847    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"
Oct 30 21:45:18.381991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.381867    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt" event={"ID":"a5ca70bb-dec2-4342-9cd9-b96be07afe88","Type":"ContainerDied","Data":"1dbf4cee6be16b5149c1b6d73d5f239f26d7891c45056520d0931ae790a92165"}
Oct 30 21:45:18.383334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.383316    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"
Oct 30 21:45:18.383412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.383335    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl" event={"ID":"b149753a-6495-4307-8b00-43ac2de2aeb4","Type":"ContainerDied","Data":"3509393abac5b5a5978d507248c7d1ac39d91fd4e6ed359d64d4a7cfabaf7b98"}
Oct 30 21:45:18.390265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.390246    6695 scope.go:122] "RemoveContainer" containerID="be3edba056155d3126168723d0bfc02faa152d7ed96d24c2a39f841f8aa8265c"
Oct 30 21:45:18.405426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.405408    6695 scope.go:122] "RemoveContainer" containerID="50d5d5a36ecb031e08f0e14cd1199d5407fc4bb6dd510e438eb63bdfd7ad5550"
Oct 30 21:45:18.410257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.410158    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"]
Oct 30 21:45:18.429983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.429945    6695 scope.go:122] "RemoveContainer" containerID="585dcc501a965ace09d05f12a1b3bb94ab6699a8992bbe72cb8b05fdfac3aa0b"
Oct 30 21:45:18.474014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.473991    6695 scope.go:122] "RemoveContainer" containerID="c42d8277a786213e13113b260f7d0a95880af9b6bdc01b90e3e0f0a81eaf0159"
Oct 30 21:45:18.514499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.514454    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"]
Oct 30 21:45:18.541536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.541504    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889945-868tt"]
Oct 30 21:45:18.548795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.548638    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"]
Oct 30 21:45:18.625188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.606256    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"]
Oct 30 21:45:18.625188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.609781    6695 scope.go:122] "RemoveContainer" containerID="7b0fa4688c71bf97b5dd214b704c33c90f62d75c4b43d94cbdd5a650ffa75515"
Oct 30 21:45:18.701073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.700909    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"]
Oct 30 21:45:18.747625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.746053    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889945-ll6fl"]
Oct 30 21:45:18.759650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.759231    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"]
Oct 30 21:45:18.767286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:18.767243    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-49738b-29889945-vdqvh"]
Oct 30 21:45:19.218730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.218692    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" path="/var/lib/kubelet/pods/52c20fd7-4f9f-4c21-a36d-d16830878d44/volumes"
Oct 30 21:45:19.219427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.219400    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" path="/var/lib/kubelet/pods/a5ca70bb-dec2-4342-9cd9-b96be07afe88/volumes"
Oct 30 21:45:19.220010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.219988    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" path="/var/lib/kubelet/pods/b149753a-6495-4307-8b00-43ac2de2aeb4/volumes"
Oct 30 21:45:19.221735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.221716    6695 scope.go:122] "RemoveContainer" containerID="26d8eb409b6602173730a22fa8e9a7099084f8bb4380d7b1d8aad26d6c9f33da"
Oct 30 21:45:19.221871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.221855    6695 scope.go:122] "RemoveContainer" containerID="b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f"
Oct 30 21:45:19.397647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.397583    6695 generic.go:356] "Generic (PLEG): container finished" podID="32b6452d-d038-404a-808f-5ff935288f7c" containerID="2c4b677bd5507e54d658ce1ebb7f6d50f78a788c9f2f668313113298e89d2e06" exitCode=60
Oct 30 21:45:19.397647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.397635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" event={"ID":"32b6452d-d038-404a-808f-5ff935288f7c","Type":"ContainerDied","Data":"2c4b677bd5507e54d658ce1ebb7f6d50f78a788c9f2f668313113298e89d2e06"}
Oct 30 21:45:19.398126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.397700    6695 scope.go:122] "RemoveContainer" containerID="2b4e86e2abbc6dc664f7547e4f53d1274440f40d3a12f458df045262856d4ac1"
Oct 30 21:45:19.398201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.398167    6695 scope.go:122] "RemoveContainer" containerID="2c4b677bd5507e54d658ce1ebb7f6d50f78a788c9f2f668313113298e89d2e06"
Oct 30 21:45:19.398680 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:19.398565    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-bshwt_odh-ai-gateway-infra(32b6452d-d038-404a-808f-5ff935288f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" podUID="32b6452d-d038-404a-808f-5ff935288f7c"
Oct 30 21:45:19.402517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.402494    6695 generic.go:356] "Generic (PLEG): container finished" podID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerID="ca91c5195adb0f0030cce8dfa43a65ecc50b3352bd8bda8322a67c0e8354bae2" exitCode=60
Oct 30 21:45:19.402670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.402555    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerDied","Data":"ca91c5195adb0f0030cce8dfa43a65ecc50b3352bd8bda8322a67c0e8354bae2"}
Oct 30 21:45:19.403229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.403212    6695 scope.go:122] "RemoveContainer" containerID="ca91c5195adb0f0030cce8dfa43a65ecc50b3352bd8bda8322a67c0e8354bae2"
Oct 30 21:45:19.403597 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:19.403575    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8_odh-ai-gateway-infra(f8fc6bcc-9a99-44bd-85c0-802f7320912a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a"
Oct 30 21:45:19.413927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.413901    6695 scope.go:122] "RemoveContainer" containerID="edfcff8495feb42100ff55d8acee1be04c89385ba2502b5668d75b8881eeaeab"
Oct 30 21:45:19.606705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.577063    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"]
Oct 30 21:45:19.705667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.705643    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:19.709625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.709591    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:19.715700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.715681    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:19.763295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763213    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca\") pod \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") "
Oct 30 21:45:19.763295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763274    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca\") pod \"436d27c5-7a02-4df3-8856-5c540f140536\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") "
Oct 30 21:45:19.763550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763385    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca\") pod \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") "
Oct 30 21:45:19.763550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763436    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27\" (UniqueName: \"kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27\") pod \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\" (UID: \"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7\") "
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763668    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca" pod "4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" (UID: "4cfbfcb8-38ac-4ebf-b990-3d482acc76c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763702    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg\" (UniqueName: \"kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg\") pod \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\" (UID: \"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4\") "
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763739    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb\" (UniqueName: \"kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb\") pod \"436d27c5-7a02-4df3-8856-5c540f140536\" (UID: \"436d27c5-7a02-4df3-8856-5c540f140536\") "
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763754    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca" pod "436d27c5-7a02-4df3-8856-5c540f140536" (UID: "436d27c5-7a02-4df3-8856-5c540f140536"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.763961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca" pod "c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" (UID: "c3c10d60-1531-4f58-a2f9-a3b129a7c4f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.764448    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.764473    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436d27c5-7a02-4df3-8856-5c540f140536-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.764486    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.766284    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27" pod "4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" (UID: "4cfbfcb8-38ac-4ebf-b990-3d482acc76c7"). InnerVolumeSpecName "kube-api-access-mfl27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.766715    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg" pod "c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" (UID: "c3c10d60-1531-4f58-a2f9-a3b129a7c4f4"). InnerVolumeSpecName "kube-api-access-48mbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.767829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.766785    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb" pod "436d27c5-7a02-4df3-8856-5c540f140536" (UID: "436d27c5-7a02-4df3-8856-5c540f140536"). InnerVolumeSpecName "kube-api-access-jz2lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.858447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.858418    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:45:19.866430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.866395    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfl27\" (UniqueName: \"kubernetes.io/projected/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7-kube-api-access-mfl27\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.866588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.866434    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48mbg\" (UniqueName: \"kubernetes.io/projected/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4-kube-api-access-48mbg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.866588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.866449    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz2lb\" (UniqueName: \"kubernetes.io/projected/436d27c5-7a02-4df3-8856-5c540f140536-kube-api-access-jz2lb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:19.869856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:19.869835    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:45:20.415569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.415534    6695 generic.go:356] "Generic (PLEG): container finished" podID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerID="aa2ec05eb95e04727097fc892fcec2798a1ff6bbb6bb77257751911485c6ed20" exitCode=60
Oct 30 21:45:20.416089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.415635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" event={"ID":"9795c21d-265d-4220-bf9e-68acade2e7ba","Type":"ContainerDied","Data":"aa2ec05eb95e04727097fc892fcec2798a1ff6bbb6bb77257751911485c6ed20"}
Oct 30 21:45:20.416089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.415698    6695 scope.go:122] "RemoveContainer" containerID="26d8eb409b6602173730a22fa8e9a7099084f8bb4380d7b1d8aad26d6c9f33da"
Oct 30 21:45:20.416210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.416171    6695 scope.go:122] "RemoveContainer" containerID="aa2ec05eb95e04727097fc892fcec2798a1ff6bbb6bb77257751911485c6ed20"
Oct 30 21:45:20.416901 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:20.416563    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2_odh-ai-gateway-infra(9795c21d-265d-4220-bf9e-68acade2e7ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba"
Oct 30 21:45:20.417388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.417368    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"
Oct 30 21:45:20.417463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.417386    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv" event={"ID":"4cfbfcb8-38ac-4ebf-b990-3d482acc76c7","Type":"ContainerDied","Data":"a5eec74dfaf26c67396cd50a896ff196342384296376793e794debf8432da963"}
Oct 30 21:45:20.419069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.419045    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls" event={"ID":"c3c10d60-1531-4f58-a2f9-a3b129a7c4f4","Type":"ContainerDied","Data":"94249991311c555de4415a3d20f077da5e077e1d929c07bad1539ff154510b15"}
Oct 30 21:45:20.419195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.419057    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"
Oct 30 21:45:20.422768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.422742    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"
Oct 30 21:45:20.422888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.422744    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6" event={"ID":"436d27c5-7a02-4df3-8856-5c540f140536","Type":"ContainerDied","Data":"adb84a0eafa2c4b858b027405cddfb22f2b382945a45759e2d760e49c527037f"}
Oct 30 21:45:20.427116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.427092    6695 generic.go:356] "Generic (PLEG): container finished" podID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerID="996ea8b7b112fa80f1b1bb037702d2bb6c62631269512d80c7fed4ee5738af5f" exitCode=60
Oct 30 21:45:20.427220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.427130    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" event={"ID":"d4c19c73-e32b-40cb-9953-52c0acc2045e","Type":"ContainerDied","Data":"996ea8b7b112fa80f1b1bb037702d2bb6c62631269512d80c7fed4ee5738af5f"}
Oct 30 21:45:20.427587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.427567    6695 scope.go:122] "RemoveContainer" containerID="996ea8b7b112fa80f1b1bb037702d2bb6c62631269512d80c7fed4ee5738af5f"
Oct 30 21:45:20.427934 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:20.427911    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb_odh-ai-gateway-infra(d4c19c73-e32b-40cb-9953-52c0acc2045e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e"
Oct 30 21:45:20.432701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.432684    6695 scope.go:122] "RemoveContainer" containerID="51fb43fb6d45ef50309328839f4c48f914c25364960149cccac5954ed24943df"
Oct 30 21:45:20.443652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.443592    6695 scope.go:122] "RemoveContainer" containerID="6ffd8a07bf5c762a05484cdabc3429cb5c9c0d1984d3b9b5e1690236c3bec92f"
Oct 30 21:45:20.458810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.457813    6695 scope.go:122] "RemoveContainer" containerID="735d973386c95cd721deffdd62c25565ae112334e547fb2449386f19b2e61393"
Oct 30 21:45:20.471442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.471420    6695 scope.go:122] "RemoveContainer" containerID="b311c94f00c175936eeb07d320dbe006f5054556adbaf518619e6874bde1c28f"
Oct 30 21:45:20.559650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.558240    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"]
Oct 30 21:45:20.563355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.563324    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889945-ms8ls"]
Oct 30 21:45:20.593020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.592990    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"]
Oct 30 21:45:20.598485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.598370    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-a2fc47-29889945-bx4hv"]
Oct 30 21:45:20.602479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.602454    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"]
Oct 30 21:45:20.609377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.609354    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:20.626309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.626285    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"]
Oct 30 21:45:20.629917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.629888    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-50ab77-29889945-448s6"]
Oct 30 21:45:20.683028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.679402    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca\") pod \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") "
Oct 30 21:45:20.683028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.679469    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf\" (UniqueName: \"kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf\") pod \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\" (UID: \"f8fc6bcc-9a99-44bd-85c0-802f7320912a\") "
Oct 30 21:45:20.683028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.681003    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca" pod "f8fc6bcc-9a99-44bd-85c0-802f7320912a" (UID: "f8fc6bcc-9a99-44bd-85c0-802f7320912a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.698839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.698806    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf" pod "f8fc6bcc-9a99-44bd-85c0-802f7320912a" (UID: "f8fc6bcc-9a99-44bd-85c0-802f7320912a"). InnerVolumeSpecName "kube-api-access-qg8xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.780450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.780426    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:20.781475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.781453    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fc6bcc-9a99-44bd-85c0-802f7320912a-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.781566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.781478    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg8xf\" (UniqueName: \"kubernetes.io/projected/f8fc6bcc-9a99-44bd-85c0-802f7320912a-kube-api-access-qg8xf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:20.921616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.920934    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g\" (UniqueName: \"kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g\") pod \"32b6452d-d038-404a-808f-5ff935288f7c\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") "
Oct 30 21:45:20.921616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.921089    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca\") pod \"32b6452d-d038-404a-808f-5ff935288f7c\" (UID: \"32b6452d-d038-404a-808f-5ff935288f7c\") "
Oct 30 21:45:20.923722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.922527    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca" pod "32b6452d-d038-404a-808f-5ff935288f7c" (UID: "32b6452d-d038-404a-808f-5ff935288f7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.963409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:20.963356    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g" pod "32b6452d-d038-404a-808f-5ff935288f7c" (UID: "32b6452d-d038-404a-808f-5ff935288f7c"). InnerVolumeSpecName "kube-api-access-ccr6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:21.024147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.024116    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b6452d-d038-404a-808f-5ff935288f7c-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:21.024147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.024142    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccr6g\" (UniqueName: \"kubernetes.io/projected/32b6452d-d038-404a-808f-5ff935288f7c-kube-api-access-ccr6g\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:21.215640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.215585    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436d27c5-7a02-4df3-8856-5c540f140536" path="/var/lib/kubelet/pods/436d27c5-7a02-4df3-8856-5c540f140536/volumes"
Oct 30 21:45:21.216206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.216187    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" path="/var/lib/kubelet/pods/4cfbfcb8-38ac-4ebf-b990-3d482acc76c7/volumes"
Oct 30 21:45:21.216717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.216699    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" path="/var/lib/kubelet/pods/c3c10d60-1531-4f58-a2f9-a3b129a7c4f4/volumes"
Oct 30 21:45:21.279868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.279798    6695 scope.go:122] "RemoveContainer" containerID="4b9251859083d9fb57bf3648112e38976227dc33f545cdcf82f438fe572c9a61"
Oct 30 21:45:21.290897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.290876    6695 scope.go:122] "RemoveContainer" containerID="0274295a512395494787220e2e8ddfcdf4b9859b7a746bda0e09b1d3b5beb4c9"
Oct 30 21:45:21.300649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.300629    6695 scope.go:122] "RemoveContainer" containerID="3233520bcf0d4092930cc3c18f186178627fe6ebea747da4543767ec4d461c15"
Oct 30 21:45:21.309992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.309971    6695 scope.go:122] "RemoveContainer" containerID="0dc4dd71a52710f9e0c018149ccdd3d15eb007398085314240b0b7332eabc55f"
Oct 30 21:45:21.319400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.319383    6695 scope.go:122] "RemoveContainer" containerID="f396ba9ea73dc06d9cd735d33ab0b937c278bc2b4fae3dc753a4564bc47cd3f2"
Oct 30 21:45:21.331850 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.331831    6695 scope.go:122] "RemoveContainer" containerID="bbde2636c1001b928852ed653165f982f4107592f17a10e7077ed3cbca09e63d"
Oct 30 21:45:21.341910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.341884    6695 scope.go:122] "RemoveContainer" containerID="871cf2815948eab3fbc9025257e7371e407fd8edf3dde80293af5cbeceb62ec8"
Oct 30 21:45:21.352403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.352385    6695 scope.go:122] "RemoveContainer" containerID="bc7cf5c009ace93444af690d6dfb13d7e4c32ef96ef14d73f73c46b948f2c7f2"
Oct 30 21:45:21.362140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.362121    6695 scope.go:122] "RemoveContainer" containerID="f445f8fac76e02b61fffb2b19fdee581a4da439f074bd4f2cef14ff14a3d302a"
Oct 30 21:45:21.374330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.374315    6695 scope.go:122] "RemoveContainer" containerID="21af11cffc85501dea2f0d41b8a1ccf4980607a227a44baa6abf93fdc51f9c5b"
Oct 30 21:45:21.385869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.385851    6695 scope.go:122] "RemoveContainer" containerID="0f3a736b4ec7321c281047598158e12504d2fef65abf788bdffba082a13ea6e4"
Oct 30 21:45:21.395151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.395132    6695 scope.go:122] "RemoveContainer" containerID="6a1f77393c35840ef966a45b9afcb7f10d1dbc36728ed740d5b6fc09f712e50e"
Oct 30 21:45:21.405193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.405173    6695 scope.go:122] "RemoveContainer" containerID="eb795d8e9fd8fce557cabbaa23276008398e108a5ed36296741c1a256b5927c0"
Oct 30 21:45:21.453885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.453848    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt" event={"ID":"32b6452d-d038-404a-808f-5ff935288f7c","Type":"ContainerDied","Data":"8a5527fdb2b462116318f5f13b934edb42446096511f2900e8b0e0e4d5ef3770"}
Oct 30 21:45:21.454249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.453859    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"
Oct 30 21:45:21.454249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.453950    6695 scope.go:122] "RemoveContainer" containerID="2c4b677bd5507e54d658ce1ebb7f6d50f78a788c9f2f668313113298e89d2e06"
Oct 30 21:45:21.461696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.461666    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8" event={"ID":"f8fc6bcc-9a99-44bd-85c0-802f7320912a","Type":"ContainerDied","Data":"3e23eeb691d289a7c087e5bbf2b4343d15f6f5fa40f48077dfcffbf5fd00a589"}
Oct 30 21:45:21.461819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.461705    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"
Oct 30 21:45:21.468584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.466377    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"]
Oct 30 21:45:21.475657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.475613    6695 scope.go:122] "RemoveContainer" containerID="ca91c5195adb0f0030cce8dfa43a65ecc50b3352bd8bda8322a67c0e8354bae2"
Oct 30 21:45:21.507004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.506975    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"]
Oct 30 21:45:21.517678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.517640    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"]
Oct 30 21:45:21.524963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.524939    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889945-gs6r8"]
Oct 30 21:45:21.559745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.559453    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"]
Oct 30 21:45:21.582227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.581417    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-bshwt"]
Oct 30 21:45:21.722129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.722102    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:21.724141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.724119    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:21.836991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.836902    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca\") pod \"9795c21d-265d-4220-bf9e-68acade2e7ba\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") "
Oct 30 21:45:21.837159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.837004    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw\" (UniqueName: \"kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw\") pod \"9795c21d-265d-4220-bf9e-68acade2e7ba\" (UID: \"9795c21d-265d-4220-bf9e-68acade2e7ba\") "
Oct 30 21:45:21.837159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.837039    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca\") pod \"d4c19c73-e32b-40cb-9953-52c0acc2045e\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") "
Oct 30 21:45:21.837293 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.837218    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76\") pod \"d4c19c73-e32b-40cb-9953-52c0acc2045e\" (UID: \"d4c19c73-e32b-40cb-9953-52c0acc2045e\") "
Oct 30 21:45:21.837360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.837288    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca" pod "9795c21d-265d-4220-bf9e-68acade2e7ba" (UID: "9795c21d-265d-4220-bf9e-68acade2e7ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:21.838087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.838058    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9795c21d-265d-4220-bf9e-68acade2e7ba-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:21.838988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.838958    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca" pod "d4c19c73-e32b-40cb-9953-52c0acc2045e" (UID: "d4c19c73-e32b-40cb-9953-52c0acc2045e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:21.841450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.841423    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw" pod "9795c21d-265d-4220-bf9e-68acade2e7ba" (UID: "9795c21d-265d-4220-bf9e-68acade2e7ba"). InnerVolumeSpecName "kube-api-access-dlvqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:21.846393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.846364    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76" pod "d4c19c73-e32b-40cb-9953-52c0acc2045e" (UID: "d4c19c73-e32b-40cb-9953-52c0acc2045e"). InnerVolumeSpecName "kube-api-access-dvc76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:21.939630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.939568    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvqw\" (UniqueName: \"kubernetes.io/projected/9795c21d-265d-4220-bf9e-68acade2e7ba-kube-api-access-dlvqw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:21.939630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.939598    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c19c73-e32b-40cb-9953-52c0acc2045e-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:21.939845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:21.939640    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/d4c19c73-e32b-40cb-9953-52c0acc2045e-kube-api-access-dvc76\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:22.534040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.534010    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"
Oct 30 21:45:22.534593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.534029    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb" event={"ID":"d4c19c73-e32b-40cb-9953-52c0acc2045e","Type":"ContainerDied","Data":"9de9a61e2cc85436fcaeb8a00e155032ab82f6144af62b919bcd7b941af535ea"}
Oct 30 21:45:22.534775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.534653    6695 scope.go:122] "RemoveContainer" containerID="996ea8b7b112fa80f1b1bb037702d2bb6c62631269512d80c7fed4ee5738af5f"
Oct 30 21:45:22.536433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.536407    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"
Oct 30 21:45:22.536669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.536588    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2" event={"ID":"9795c21d-265d-4220-bf9e-68acade2e7ba","Type":"ContainerDied","Data":"fc464f71d313c7c133508aae74a17dca7779a2b425740b607fd0ec6ba6d49bdb"}
Oct 30 21:45:22.548790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.548767    6695 scope.go:122] "RemoveContainer" containerID="aa2ec05eb95e04727097fc892fcec2798a1ff6bbb6bb77257751911485c6ed20"
Oct 30 21:45:22.588939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.588904    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"]
Oct 30 21:45:22.598692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.598637    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889945-lfzcb"]
Oct 30 21:45:22.630172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.630144    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"]
Oct 30 21:45:22.633892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.633865    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889945-xggw2"]
Oct 30 21:45:22.847026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.846936    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52" podUID="8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" containerName="payload-processing" containerID="cri-o://41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b" gracePeriod=25
Oct 30 21:45:22.884400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:22.884368    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt" podUID="2c92eaaf-658a-423d-8b16-7cfebc93e7b1" containerName="payload-pre-processing" containerID="cri-o://665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33" gracePeriod=25
Oct 30 21:45:23.172267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.172237    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:45:23.214918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.214845    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:45:23.219535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.219509    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b6452d-d038-404a-808f-5ff935288f7c" path="/var/lib/kubelet/pods/32b6452d-d038-404a-808f-5ff935288f7c/volumes"
Oct 30 21:45:23.220043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.220027    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" path="/var/lib/kubelet/pods/9795c21d-265d-4220-bf9e-68acade2e7ba/volumes"
Oct 30 21:45:23.220498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.220485    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" path="/var/lib/kubelet/pods/d4c19c73-e32b-40cb-9953-52c0acc2045e/volumes"
Oct 30 21:45:23.221344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.221323    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" path="/var/lib/kubelet/pods/f8fc6bcc-9a99-44bd-85c0-802f7320912a/volumes"
Oct 30 21:45:23.262157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.262114    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume\") pod \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") "
Oct 30 21:45:23.262386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.262363    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p\" (UniqueName: \"kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p\") pod \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") "
Oct 30 21:45:23.262473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.262429    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume\") pod \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\" (UID: \"2c92eaaf-658a-423d-8b16-7cfebc93e7b1\") "
Oct 30 21:45:23.262543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.262477    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4\" (UniqueName: \"kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4\") pod \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\" (UID: \"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58\") "
Oct 30 21:45:23.262543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.262497    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume" pod "8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" (UID: "8d71b7ba-dcc3-4d78-bc95-52b4c0116e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:23.265782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.262977    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume" pod "2c92eaaf-658a-423d-8b16-7cfebc93e7b1" (UID: "2c92eaaf-658a-423d-8b16-7cfebc93e7b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:23.265782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.263218    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:23.265782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.263240    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:23.265782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.265102    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p" pod "2c92eaaf-658a-423d-8b16-7cfebc93e7b1" (UID: "2c92eaaf-658a-423d-8b16-7cfebc93e7b1"). InnerVolumeSpecName "kube-api-access-42s8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:23.265782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.265684    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4" pod "8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" (UID: "8d71b7ba-dcc3-4d78-bc95-52b4c0116e58"). InnerVolumeSpecName "kube-api-access-tk2w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:23.364692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.364655    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42s8p\" (UniqueName: \"kubernetes.io/projected/2c92eaaf-658a-423d-8b16-7cfebc93e7b1-kube-api-access-42s8p\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:23.364692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.364684    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2w4\" (UniqueName: \"kubernetes.io/projected/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58-kube-api-access-tk2w4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:23.549059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.549020    6695 generic.go:356] "Generic (PLEG): container finished" podID="2c92eaaf-658a-423d-8b16-7cfebc93e7b1" containerID="665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33" exitCode=0
Oct 30 21:45:23.549537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.549073    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"
Oct 30 21:45:23.549537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.549095    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt" event={"ID":"2c92eaaf-658a-423d-8b16-7cfebc93e7b1","Type":"ContainerDied","Data":"665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33"}
Oct 30 21:45:23.549537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.549153    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt" event={"ID":"2c92eaaf-658a-423d-8b16-7cfebc93e7b1","Type":"ContainerDied","Data":"407b3a9f6cdb17af3bc2c27fb8d82725a3c8cce68603188168f7f0213ddf1eda"}
Oct 30 21:45:23.549537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.549167    6695 scope.go:122] "RemoveContainer" containerID="665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33"
Oct 30 21:45:23.550755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.550725    6695 generic.go:356] "Generic (PLEG): container finished" podID="8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" containerID="41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b" exitCode=0
Oct 30 21:45:23.550853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.550777    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"
Oct 30 21:45:23.550906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.550770    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52" event={"ID":"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58","Type":"ContainerDied","Data":"41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b"}
Oct 30 21:45:23.550951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.550919    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52" event={"ID":"8d71b7ba-dcc3-4d78-bc95-52b4c0116e58","Type":"ContainerDied","Data":"5f09e860f14b5a00205366c88a79e59e3b869b8226da5ed7dc6130bee9e5f6d7"}
Oct 30 21:45:23.567184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.567166    6695 scope.go:122] "RemoveContainer" containerID="665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33"
Oct 30 21:45:23.567530 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:23.567510    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33\": container with ID starting with 665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33 not found: ID does not exist" containerID="665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33"
Oct 30 21:45:23.567660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.567541    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33"} err="failed to get container status \"665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33\": rpc error: code = NotFound desc = could not find container \"665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33\": container with ID starting with 665e8871dd6df29bbbf09ade7925a7450705ba084d04c998b89f62898d8b4d33 not found: ID does not exist"
Oct 30 21:45:23.567660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.567564    6695 scope.go:122] "RemoveContainer" containerID="41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b"
Oct 30 21:45:23.579087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.579066    6695 scope.go:122] "RemoveContainer" containerID="41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b"
Oct 30 21:45:23.579411 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:23.579391    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b\": container with ID starting with 41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b not found: ID does not exist" containerID="41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b"
Oct 30 21:45:23.579474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.579423    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b"} err="failed to get container status \"41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b\": rpc error: code = NotFound desc = could not find container \"41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b\": container with ID starting with 41bb5a02c9bcc1cb593864192131e9ab1b6b88405599100ca09f03ca2a30f90b not found: ID does not exist"
Oct 30 21:45:23.614528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.614499    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"]
Oct 30 21:45:23.625573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.625548    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a2fc47-6586454b55-p98kt"]
Oct 30 21:45:23.657385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.657354    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"]
Oct 30 21:45:23.663053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:23.663028    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a2fc47-6d9866d987-wxq52"]
Oct 30 21:45:25.216723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:25.216689    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c92eaaf-658a-423d-8b16-7cfebc93e7b1" path="/var/lib/kubelet/pods/2c92eaaf-658a-423d-8b16-7cfebc93e7b1/volumes"
Oct 30 21:45:25.217304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:25.217288    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" path="/var/lib/kubelet/pods/8d71b7ba-dcc3-4d78-bc95-52b4c0116e58/volumes"
Oct 30 21:45:29.041463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:29.041429    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"]
Oct 30 21:45:29.053642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:29.053595    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-808058f8-fbbjn"]
Oct 30 21:45:29.217198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:29.217165    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7b2e82-ae7b-494d-8e54-fc4b9be48579" path="/var/lib/kubelet/pods/fa7b2e82-ae7b-494d-8e54-fc4b9be48579/volumes"
Oct 30 21:45:31.073801 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:31.073764    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:45:37.138896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.138860    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"]
Oct 30 21:45:37.139563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139547    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" containerName="payload-processing"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139566    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" containerName="payload-processing"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139583    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139591    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139610    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139616    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139626    6695 cpu_manager.go:414] "containerMap: removing container" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139630    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139636    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139641    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139650    6695 cpu_manager.go:414] "containerMap: removing container" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139654    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139660    6695 cpu_manager.go:414] "containerMap: removing container" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139664    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139672    6695 cpu_manager.go:414] "containerMap: removing container" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:37.139670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139676    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139690    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139694    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139703    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139707    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139713    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139718    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139727    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139731    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139737    6695 cpu_manager.go:414] "containerMap: removing container" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139741    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139747    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139751    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139759    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139763    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139772    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139776    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139783    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139787    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139795    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139799    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139804    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139808    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139816    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139820    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139834    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139839    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139849    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2c92eaaf-658a-423d-8b16-7cfebc93e7b1" containerName="payload-pre-processing"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139854    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c92eaaf-658a-423d-8b16-7cfebc93e7b1" containerName="payload-pre-processing"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139866    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c0d1e4dd-76e1-4c25-b446-4232be74a130" containerName="revoke-keys"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139870    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d1e4dd-76e1-4c25-b446-4232be74a130" containerName="revoke-keys"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139882    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139887    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139894    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139899    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139910    6695 cpu_manager.go:414] "containerMap: removing container" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.139915    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140030    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d1e4dd-76e1-4c25-b446-4232be74a130" containerName="revoke-keys"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140039    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140044    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140051    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140055    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140060    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140066    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140071    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140075    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140081    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d71b7ba-dcc3-4d78-bc95-52b4c0116e58" containerName="payload-processing"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140088    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140094    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140099    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140105    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140110    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140116    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140124    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140130    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140137    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140143    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140149    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.140130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140156    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:37.142417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140161    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:37.142417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140167    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:37.142417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140171    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfbfcb8-38ac-4ebf-b990-3d482acc76c7" containerName="cleanup"
Oct 30 21:45:37.142417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140176    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:37.142417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140184    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c19c73-e32b-40cb-9953-52c0acc2045e" containerName="cleanup"
Oct 30 21:45:37.142417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.140189    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c92eaaf-658a-423d-8b16-7cfebc93e7b1" containerName="payload-pre-processing"
Oct 30 21:45:37.144035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.144020    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.163006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.162969    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"]
Oct 30 21:45:37.169887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.169863    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:45:37.169887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.169869    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:45:37.170100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.169918    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:45:37.170100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.169947    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:45:37.262365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.262329    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.262567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.262404    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.262567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.262522    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wwt\" (UniqueName: \"kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.366835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.365753    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.366835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.365860    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.366835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.365949    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wwt\" (UniqueName: \"kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.367810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.367779    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.369680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.369657    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.386233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.386210    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wwt\" (UniqueName: \"kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt\") pod \"maas-api-revoke-sub-b21fa274-n6lf4\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.458943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.458868    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:37.764474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:37.764447    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"]
Oct 30 21:45:37.767024 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:37.766988    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26b965c_48a7_4920_96aa_c7dab4bbfefa.slice/crio-ad1ec122f752bb016f9d5172c63626e77f18e7236c6ff7cdd12e3daa024cdaf3 WatchSource:0}: Error finding container ad1ec122f752bb016f9d5172c63626e77f18e7236c6ff7cdd12e3daa024cdaf3: Status 404 returned error can't find the container with id ad1ec122f752bb016f9d5172c63626e77f18e7236c6ff7cdd12e3daa024cdaf3
Oct 30 21:45:38.782371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:38.782333    6695 generic.go:356] "Generic (PLEG): container finished" podID="e26b965c-48a7-4920-96aa-c7dab4bbfefa" containerID="ac9ddc0bb49507308c0a498dc655ab7a506064f58e5734303430bd69d7f1d93e" exitCode=0
Oct 30 21:45:38.782775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:38.782427    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4" event={"ID":"e26b965c-48a7-4920-96aa-c7dab4bbfefa","Type":"ContainerDied","Data":"ac9ddc0bb49507308c0a498dc655ab7a506064f58e5734303430bd69d7f1d93e"}
Oct 30 21:45:38.782775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:38.782480    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4" event={"ID":"e26b965c-48a7-4920-96aa-c7dab4bbfefa","Type":"ContainerStarted","Data":"ad1ec122f752bb016f9d5172c63626e77f18e7236c6ff7cdd12e3daa024cdaf3"}
Oct 30 21:45:39.967522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:39.967494    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:40.127169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.127076    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt\" (UniqueName: \"kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt\") pod \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") "
Oct 30 21:45:40.127346 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.127194    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca\") pod \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") "
Oct 30 21:45:40.127432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.127404    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca\") pod \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\" (UID: \"e26b965c-48a7-4920-96aa-c7dab4bbfefa\") "
Oct 30 21:45:40.127833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.127798    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca" pod "e26b965c-48a7-4920-96aa-c7dab4bbfefa" (UID: "e26b965c-48a7-4920-96aa-c7dab4bbfefa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:40.128340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.128317    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:40.129407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.129380    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt" pod "e26b965c-48a7-4920-96aa-c7dab4bbfefa" (UID: "e26b965c-48a7-4920-96aa-c7dab4bbfefa"). InnerVolumeSpecName "kube-api-access-w2wwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:40.129563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.129537    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca" pod "e26b965c-48a7-4920-96aa-c7dab4bbfefa" (UID: "e26b965c-48a7-4920-96aa-c7dab4bbfefa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:40.230084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.230053    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wwt\" (UniqueName: \"kubernetes.io/projected/e26b965c-48a7-4920-96aa-c7dab4bbfefa-kube-api-access-w2wwt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:40.230084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.230083    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26b965c-48a7-4920-96aa-c7dab4bbfefa-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:40.900408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.900375    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"
Oct 30 21:45:40.900688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.900595    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4" event={"ID":"e26b965c-48a7-4920-96aa-c7dab4bbfefa","Type":"ContainerDied","Data":"ad1ec122f752bb016f9d5172c63626e77f18e7236c6ff7cdd12e3daa024cdaf3"}
Oct 30 21:45:40.900688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:40.900676    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad1ec122f752bb016f9d5172c63626e77f18e7236c6ff7cdd12e3daa024cdaf3"
Oct 30 21:45:41.158746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:41.158656    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"]
Oct 30 21:45:41.162504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:41.162476    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a2fc47-5bsc9"]
Oct 30 21:45:41.271900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:41.271583    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d1e4dd-76e1-4c25-b446-4232be74a130" path="/var/lib/kubelet/pods/c0d1e4dd-76e1-4c25-b446-4232be74a130/volumes"
Oct 30 21:45:44.073914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:44.073875    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"]
Oct 30 21:45:44.074614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:44.074114    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy" containerID="cri-o://cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d" gracePeriod=30
Oct 30 21:45:44.078029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:44.078000    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"]
Oct 30 21:45:44.078200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:44.078181    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy" containerID="cri-o://c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240" gracePeriod=30
Oct 30 21:45:44.084518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:44.084491    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"]
Oct 30 21:45:44.084705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:44.084684    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy" containerID="cri-o://465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb" gracePeriod=30
Oct 30 21:45:47.471817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.471783    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"]
Oct 30 21:45:47.472489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472475    6695 cpu_manager.go:414] "containerMap: removing container" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:47.472535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472490    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:47.472535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472500    6695 cpu_manager.go:414] "containerMap: removing container" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:47.472535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472507    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:47.472535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472528    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:47.472535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472533    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c10d60-1531-4f58-a2f9-a3b129a7c4f4" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472543    6695 cpu_manager.go:414] "containerMap: removing container" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472548    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c20fd7-4f9f-4c21-a36d-d16830878d44" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472562    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472567    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472575    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472580    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472588    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472593    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fc6bcc-9a99-44bd-85c0-802f7320912a" containerName="cleanup"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472617    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e26b965c-48a7-4920-96aa-c7dab4bbfefa" containerName="revoke-keys"
Oct 30 21:45:47.472726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472624    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26b965c-48a7-4920-96aa-c7dab4bbfefa" containerName="revoke-keys"
Oct 30 21:45:47.473043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472771    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="436d27c5-7a02-4df3-8856-5c540f140536" containerName="cleanup"
Oct 30 21:45:47.473043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472784    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b6452d-d038-404a-808f-5ff935288f7c" containerName="cleanup"
Oct 30 21:45:47.473043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472795    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9795c21d-265d-4220-bf9e-68acade2e7ba" containerName="cleanup"
Oct 30 21:45:47.473043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472802    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ca70bb-dec2-4342-9cd9-b96be07afe88" containerName="cleanup"
Oct 30 21:45:47.473043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472810    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26b965c-48a7-4920-96aa-c7dab4bbfefa" containerName="revoke-keys"
Oct 30 21:45:47.473043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.472821    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b149753a-6495-4307-8b00-43ac2de2aeb4" containerName="cleanup"
Oct 30 21:45:47.476266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.476250    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.491764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.491740    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:45:47.491878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.491784    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:45:47.502930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.502906    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:45:47.503058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.502998    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:45:47.507700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.507678    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"]
Oct 30 21:45:47.629631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.612955    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.629631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.613013    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmnpn\" (UniqueName: \"kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.629631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.613444    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.715260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.715221    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.715437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.715364    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.715437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.715391    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnpn\" (UniqueName: \"kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.715916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.715896    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.717798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.717773    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.739180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.739106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmnpn\" (UniqueName: \"kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn\") pod \"maas-api-revoke-sub-2e8183a2-9s7vw\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:47.791656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:47.791390    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:48.286032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:48.284045    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"]
Oct 30 21:45:48.293052 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:48.293017    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431b2737_eddb_4c5a_90a8_7d7fa87b0fb0.slice/crio-ae00548906303c16410f88d8a52d8b630b08a16ae28925746730163efb158b0f WatchSource:0}: Error finding container ae00548906303c16410f88d8a52d8b630b08a16ae28925746730163efb158b0f: Status 404 returned error can't find the container with id ae00548906303c16410f88d8a52d8b630b08a16ae28925746730163efb158b0f
Oct 30 21:45:48.974664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:48.974567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw" event={"ID":"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0","Type":"ContainerStarted","Data":"7634c6a5b71bb4f560065f0a431401982582914e0209292ecccb8f3408108b21"}
Oct 30 21:45:48.974664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:48.974653    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw" event={"ID":"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0","Type":"ContainerStarted","Data":"ae00548906303c16410f88d8a52d8b630b08a16ae28925746730163efb158b0f"}
Oct 30 21:45:49.015831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.015781    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw" podStartSLOduration=2.015739175 podStartE2EDuration="2.015739175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:49.014543578 +0000 UTC m=+2131.937001877" watchObservedRunningTime="2026-10-30 21:45:49.015739175 +0000 UTC m=+2131.938197472"
Oct 30 21:45:49.663810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.663780    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:45:49.774073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774040    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:45:49.774247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774094    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774154    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774234    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774291    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774334    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774398    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774467 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774449    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.774778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774488    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774578    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774657    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data\") pod \"98fbc84f-5818-40c6-ae6a-04c205d108ee\" (UID: \"98fbc84f-5818-40c6-ae6a-04c205d108ee\") "
Oct 30 21:45:49.774778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.774722    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.775045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775025    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:49.775279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775253    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.775641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775590    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.775963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775941    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.776055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775969    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.776055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775983    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.776055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.775998    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fbc84f-5818-40c6-ae6a-04c205d108ee-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.776055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.776011    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.778583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.778562    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.778721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.778670    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:49.778844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.778822    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "kube-api-access-thjzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:49.779922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.779895    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo" pod "98fbc84f-5818-40c6-ae6a-04c205d108ee" (UID: "98fbc84f-5818-40c6-ae6a-04c205d108ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:45:49.780811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.780793    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.876786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.876842    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.876903    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.876946    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.876992    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877054    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877094    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877127    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877156    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877179    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877261    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877338    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877377    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877382    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877463    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877501    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy\") pod \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\" (UID: \"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877536    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877568    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877639    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket\") pod \"722b7290-7400-4077-89a8-c30112abc9cb\" (UID: \"722b7290-7400-4077-89a8-c30112abc9cb\") "
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877705    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877778    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877874    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.877956    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878040    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878172    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878413    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878429    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878438    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878450    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878462    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b7290-7400-4077-89a8-c30112abc9cb-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878470    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878478    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878487    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thjzg\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-kube-api-access-thjzg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878499    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878511    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878519    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.878527    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fbc84f-5818-40c6-ae6a-04c205d108ee-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.879272    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.879644    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.879887    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.880867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.880887    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:49.883644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.881167    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.883644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.882567    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:49.883644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.882697    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:45:49.884337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.884293    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "kube-api-access-fsgb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:49.884587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.884565    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns" pod "722b7290-7400-4077-89a8-c30112abc9cb" (UID: "722b7290-7400-4077-89a8-c30112abc9cb"). InnerVolumeSpecName "kube-api-access-z8wns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:49.887570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.887541    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy" pod "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" (UID: "21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:49.980242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980209    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980251    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980260    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980270    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wns\" (UniqueName: \"kubernetes.io/projected/722b7290-7400-4077-89a8-c30112abc9cb-kube-api-access-z8wns\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980280    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsgb4\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-kube-api-access-fsgb4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980289    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980301    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980312    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980320    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b7290-7400-4077-89a8-c30112abc9cb-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.980766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.980328    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b7290-7400-4077-89a8-c30112abc9cb-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:49.985174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.985146    6695 generic.go:356] "Generic (PLEG): container finished" podID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerID="cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d" exitCode=0
Oct 30 21:45:49.985325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.985194    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"
Oct 30 21:45:49.985325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.985242    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" event={"ID":"98fbc84f-5818-40c6-ae6a-04c205d108ee","Type":"ContainerDied","Data":"cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d"}
Oct 30 21:45:49.985325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.985305    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm" event={"ID":"98fbc84f-5818-40c6-ae6a-04c205d108ee","Type":"ContainerDied","Data":"06b56e7a2ed1e6fd8a2c1f2627d4fd05ee447a691a60bcb24c729d7e60ab1f84"}
Oct 30 21:45:49.985497 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.985335    6695 scope.go:122] "RemoveContainer" containerID="cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d"
Oct 30 21:45:49.986993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.986970    6695 generic.go:356] "Generic (PLEG): container finished" podID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerID="465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb" exitCode=0
Oct 30 21:45:49.987183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.987035    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"
Oct 30 21:45:49.987183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.987061    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" event={"ID":"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6","Type":"ContainerDied","Data":"465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb"}
Oct 30 21:45:49.987183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.987136    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j" event={"ID":"21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6","Type":"ContainerDied","Data":"ba24d2375f9b8aab6f118b9d9d0ea00967c31018223dbd6f4ee5c7206f652b27"}
Oct 30 21:45:49.988890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.988866    6695 generic.go:356] "Generic (PLEG): container finished" podID="722b7290-7400-4077-89a8-c30112abc9cb" containerID="c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240" exitCode=0
Oct 30 21:45:49.989024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.988938    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"
Oct 30 21:45:49.989024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.988975    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" event={"ID":"722b7290-7400-4077-89a8-c30112abc9cb","Type":"ContainerDied","Data":"c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240"}
Oct 30 21:45:49.989544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.989054    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv" event={"ID":"722b7290-7400-4077-89a8-c30112abc9cb","Type":"ContainerDied","Data":"d6e9e8e0e2684a26b142844b7d7b15ff5a0402289ad381907aea4c35ab7edc8d"}
Oct 30 21:45:49.991013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.990938    6695 generic.go:356] "Generic (PLEG): container finished" podID="431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" containerID="7634c6a5b71bb4f560065f0a431401982582914e0209292ecccb8f3408108b21" exitCode=0
Oct 30 21:45:49.991013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.990972    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw" event={"ID":"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0","Type":"ContainerDied","Data":"7634c6a5b71bb4f560065f0a431401982582914e0209292ecccb8f3408108b21"}
Oct 30 21:45:49.999037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.998996    6695 scope.go:122] "RemoveContainer" containerID="cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d"
Oct 30 21:45:49.999432 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:49.999367    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d\": container with ID starting with cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d not found: ID does not exist" containerID="cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d"
Oct 30 21:45:49.999432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.999402    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d"} err="failed to get container status \"cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d\": rpc error: code = NotFound desc = could not find container \"cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d\": container with ID starting with cc1ae0142ff3484fce9285b7269d52139b658e0132aa586b9e1b3481f7f8432d not found: ID does not exist"
Oct 30 21:45:49.999432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:49.999426    6695 scope.go:122] "RemoveContainer" containerID="465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb"
Oct 30 21:45:50.012419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.012401    6695 scope.go:122] "RemoveContainer" containerID="465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb"
Oct 30 21:45:50.012753 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:50.012735    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb\": container with ID starting with 465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb not found: ID does not exist" containerID="465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb"
Oct 30 21:45:50.012824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.012760    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb"} err="failed to get container status \"465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb\": rpc error: code = NotFound desc = could not find container \"465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb\": container with ID starting with 465a54fdebf776b80c2d47c82480cbb9a1d8b99b8d4bc761243c70ae4f3e15cb not found: ID does not exist"
Oct 30 21:45:50.012824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.012778    6695 scope.go:122] "RemoveContainer" containerID="c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240"
Oct 30 21:45:50.022630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.022609    6695 scope.go:122] "RemoveContainer" containerID="c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240"
Oct 30 21:45:50.022909 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:45:50.022892    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240\": container with ID starting with c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240 not found: ID does not exist" containerID="c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240"
Oct 30 21:45:50.022953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.022915    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240"} err="failed to get container status \"c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240\": rpc error: code = NotFound desc = could not find container \"c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240\": container with ID starting with c82398ebe78a7075f03dfc2b855aa012a770695dc774b9ee4f5561a014f1c240 not found: ID does not exist"
Oct 30 21:45:50.051764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.051729    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"]
Oct 30 21:45:50.057495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.057464    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-5jcwm"]
Oct 30 21:45:50.142298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.142204    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"]
Oct 30 21:45:50.154024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.153989    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-vq68j"]
Oct 30 21:45:50.187422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.187389    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"]
Oct 30 21:45:50.192210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:50.192182    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-zgmtv"]
Oct 30 21:45:51.174822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.174795    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:51.231824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.230500    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn\" (UniqueName: \"kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn\") pod \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") "
Oct 30 21:45:51.231824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.230861    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca\") pod \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") "
Oct 30 21:45:51.231824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.230935    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca\") pod \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\" (UID: \"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0\") "
Oct 30 21:45:51.232637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.232305    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca" pod "431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" (UID: "431b2737-eddb-4c5a-90a8-7d7fa87b0fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:51.248214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.248163    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca" pod "431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" (UID: "431b2737-eddb-4c5a-90a8-7d7fa87b0fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:51.253393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.252870    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" path="/var/lib/kubelet/pods/21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6/volumes"
Oct 30 21:45:51.300038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.253899    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722b7290-7400-4077-89a8-c30112abc9cb" path="/var/lib/kubelet/pods/722b7290-7400-4077-89a8-c30112abc9cb/volumes"
Oct 30 21:45:51.300038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.258922    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" path="/var/lib/kubelet/pods/98fbc84f-5818-40c6-ae6a-04c205d108ee/volumes"
Oct 30 21:45:51.347621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.342769    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:51.347621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.342800    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:51.378767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.374875    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn" pod "431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" (UID: "431b2737-eddb-4c5a-90a8-7d7fa87b0fb0"). InnerVolumeSpecName "kube-api-access-jmnpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:51.447292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:51.447087    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmnpn\" (UniqueName: \"kubernetes.io/projected/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0-kube-api-access-jmnpn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:45:52.010406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.010369    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw" event={"ID":"431b2737-eddb-4c5a-90a8-7d7fa87b0fb0","Type":"ContainerDied","Data":"ae00548906303c16410f88d8a52d8b630b08a16ae28925746730163efb158b0f"}
Oct 30 21:45:52.010591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.010412    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"
Oct 30 21:45:52.010591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.010433    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae00548906303c16410f88d8a52d8b630b08a16ae28925746730163efb158b0f"
Oct 30 21:45:52.409453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.409414    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"]
Oct 30 21:45:52.410623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410580    6695 cpu_manager.go:414] "containerMap: removing container" podUID="431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" containerName="revoke-keys"
Oct 30 21:45:52.410774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410747    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" containerName="revoke-keys"
Oct 30 21:45:52.410846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410788    6695 cpu_manager.go:414] "containerMap: removing container" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy"
Oct 30 21:45:52.410846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410796    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy"
Oct 30 21:45:52.410846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410827    6695 cpu_manager.go:414] "containerMap: removing container" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy"
Oct 30 21:45:52.410846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410834    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy"
Oct 30 21:45:52.411059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410863    6695 cpu_manager.go:414] "containerMap: removing container" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy"
Oct 30 21:45:52.411059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.410871    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy"
Oct 30 21:45:52.411169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.411055    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" containerName="revoke-keys"
Oct 30 21:45:52.411169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.411076    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e413a6-fdf1-45ca-ba7c-f4d051d9dfd6" containerName="istio-proxy"
Oct 30 21:45:52.411169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.411089    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="722b7290-7400-4077-89a8-c30112abc9cb" containerName="istio-proxy"
Oct 30 21:45:52.411169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.411100    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fbc84f-5818-40c6-ae6a-04c205d108ee" containerName="istio-proxy"
Oct 30 21:45:52.416930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.416904    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.435407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.435379    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"]
Oct 30 21:45:52.441305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.441269    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0d74b3\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.441456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.441318    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0d74b3\"/\"default-dockercfg-vnr6f\""
Oct 30 21:45:52.442558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.442536    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0d74b3\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.442723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.442663    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0d74b3\"/\"noten-model-0d74b3-kserve-self-signed-certs\""
Oct 30 21:45:52.569523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.569483    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h827c\" (UniqueName: \"kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.569735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.569639    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.569735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.569698    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.569862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.569823    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.569917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.569900    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.569961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.569949    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.681350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.681250    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h827c\" (UniqueName: \"kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.681735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.681708    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.681949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.681918    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.682267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.682238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.682428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.682407    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.682635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.682588    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.682826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.682806    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.710629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.683430    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.710629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.683460    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.726625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.723379    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.726625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.723689    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:52.752633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:52.750345    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h827c\" (UniqueName: \"kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c\") pod \"noten-model-0d74b3-kserve-5fd8dfb757-vr2sw\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") " pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:53.032243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:53.032208    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:45:53.341888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:53.341699    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"]
Oct 30 21:45:53.344856 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:45:53.344813    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4ddea4c_22e3_4a2f_b1fc_9165b5ab85c1.slice/crio-327a81aeb22f8ec4ed153de8ace1fffd69b22807614653f3de4c38cdedb956f4 WatchSource:0}: Error finding container 327a81aeb22f8ec4ed153de8ace1fffd69b22807614653f3de4c38cdedb956f4: Status 404 returned error can't find the container with id 327a81aeb22f8ec4ed153de8ace1fffd69b22807614653f3de4c38cdedb956f4
Oct 30 21:45:54.037613 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:54.037566    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw" event={"ID":"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1","Type":"ContainerStarted","Data":"87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a"}
Oct 30 21:45:54.038155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:54.037650    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw" event={"ID":"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1","Type":"ContainerStarted","Data":"327a81aeb22f8ec4ed153de8ace1fffd69b22807614653f3de4c38cdedb956f4"}
Oct 30 21:45:54.077364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:54.077301    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw" podStartSLOduration=2.077286933 podStartE2EDuration="2.077286933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:54.076649103 +0000 UTC m=+2136.999107400" watchObservedRunningTime="2026-10-30 21:45:54.077286933 +0000 UTC m=+2136.999745230"
Oct 30 21:45:55.033378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:45:55.033341    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:46:01.075001 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:01.074964    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:46:01.584093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.584058    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"]
Oct 30 21:46:01.584332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.584305    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy" containerID="cri-o://ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc" gracePeriod=30
Oct 30 21:46:01.596254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.596216    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"]
Oct 30 21:46:01.596890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.596831    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy" containerID="cri-o://ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8" gracePeriod=30
Oct 30 21:46:01.599664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.599639    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"]
Oct 30 21:46:01.599927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.599846    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy" containerID="cri-o://2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213" gracePeriod=30
Oct 30 21:46:01.607147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.607115    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"]
Oct 30 21:46:01.607362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.607340    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" containerID="cri-o://54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2" gracePeriod=30
Oct 30 21:46:01.689853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.689815    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:46:01.690026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:01.689878    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:46:03.032691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:03.032643    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:46:03.045732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:03.045701    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:46:03.127864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:03.127834    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:46:06.930571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:06.930397    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:46:06.973217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:06.973185    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:46:06.977562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:06.977530    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:46:07.014444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.014412    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:46:07.051273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051233    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.051447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051295    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.051447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051331    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.051447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051347    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.051447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051370    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.051447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051426    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.051779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051468    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.051779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051494    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.051779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051499    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.051779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051639    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.051779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051719    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051791    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051821    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051898    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051940    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051936    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051973    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051981    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.051990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052013    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.052117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052114    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052167    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052167    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052212    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052221    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052252    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052280    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052313    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052345    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052338    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052376    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo\") pod \"2190db3a-850e-4ca5-a71b-436879f0988a\" (UID: \"2190db3a-850e-4ca5-a71b-436879f0988a\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052452    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052524    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy\") pod \"ea3a6a78-9914-46c0-b85b-6efb55452ade\" (UID: \"ea3a6a78-9914-46c0-b85b-6efb55452ade\") "
Oct 30 21:46:07.052703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052559    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo\") pod \"a287432e-f634-4fbd-bb68-7e04f867f6c2\" (UID: \"a287432e-f634-4fbd-bb68-7e04f867f6c2\") "
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053380    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053404    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053418    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053432    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053445    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053458    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053471    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.053484    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.053980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052558    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.053980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.052575    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:07.054180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.054155    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.054313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.054289    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.054447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.054311    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:07.054519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.054500    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.054809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.054782    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:07.055359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.055317    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.057327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.057300    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:07.057445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.057350    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.058167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058089    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "kube-api-access-p722x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.058167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058120    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.058167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058133    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:07.058547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058520    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.058700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058639    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.058700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058672    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "kube-api-access-wcm72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.058815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.058714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29" pod "ea3a6a78-9914-46c0-b85b-6efb55452ade" (UID: "ea3a6a78-9914-46c0-b85b-6efb55452ade"). InnerVolumeSpecName "kube-api-access-59r29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.059127 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.059107    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy" pod "a287432e-f634-4fbd-bb68-7e04f867f6c2" (UID: "a287432e-f634-4fbd-bb68-7e04f867f6c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.060053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.060036    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo" pod "2190db3a-850e-4ca5-a71b-436879f0988a" (UID: "2190db3a-850e-4ca5-a71b-436879f0988a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:07.152885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.152778    6695 generic.go:356] "Generic (PLEG): container finished" podID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerID="ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8" exitCode=0
Oct 30 21:46:07.152885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.152821    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" event={"ID":"ea3a6a78-9914-46c0-b85b-6efb55452ade","Type":"ContainerDied","Data":"ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8"}
Oct 30 21:46:07.152885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.152842    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"
Oct 30 21:46:07.153157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.152970    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm" event={"ID":"ea3a6a78-9914-46c0-b85b-6efb55452ade","Type":"ContainerDied","Data":"51daf71d6ae67c916f138d616f673a86c71666c51c5265dc47580f5d39d7a328"}
Oct 30 21:46:07.153157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.153007    6695 scope.go:122] "RemoveContainer" containerID="ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8"
Oct 30 21:46:07.153978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.153942    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154001    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154092    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154163    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154251    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154280    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.154416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154332    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154381    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154433    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154572    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data\") pod \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\" (UID: \"a711ee9f-5969-4295-92a6-c0e6a560a8e4\") "
Oct 30 21:46:07.154757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154656    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:07.155181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154923    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.155181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.154943    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.155181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155131    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155254    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcm72\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-kube-api-access-wcm72\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155268    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155277    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155320    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155334    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2190db3a-850e-4ca5-a71b-436879f0988a-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155348    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155361    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2190db3a-850e-4ca5-a71b-436879f0988a-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155373    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155387    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155401    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155413    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.155426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155425    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155439    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155453    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a287432e-f634-4fbd-bb68-7e04f867f6c2-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155464    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155477    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a287432e-f634-4fbd-bb68-7e04f867f6c2-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155490    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p722x\" (UniqueName: \"kubernetes.io/projected/a287432e-f634-4fbd-bb68-7e04f867f6c2-kube-api-access-p722x\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155504    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2190db3a-850e-4ca5-a71b-436879f0988a-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155532    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3a6a78-9914-46c0-b85b-6efb55452ade-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155546    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2190db3a-850e-4ca5-a71b-436879f0988a-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155559    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59r29\" (UniqueName: \"kubernetes.io/projected/ea3a6a78-9914-46c0-b85b-6efb55452ade-kube-api-access-59r29\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155572    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3a6a78-9914-46c0-b85b-6efb55452ade-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155585    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a287432e-f634-4fbd-bb68-7e04f867f6c2-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.155596    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.156804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.156749    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.157235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.157206    6695 generic.go:356] "Generic (PLEG): container finished" podID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerID="ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc" exitCode=0
Oct 30 21:46:07.157365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.157310    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "kube-api-access-vqk6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:07.157447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.157369    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" event={"ID":"a287432e-f634-4fbd-bb68-7e04f867f6c2","Type":"ContainerDied","Data":"ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc"}
Oct 30 21:46:07.157447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.157405    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"
Oct 30 21:46:07.157447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.157421    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp" event={"ID":"a287432e-f634-4fbd-bb68-7e04f867f6c2","Type":"ContainerDied","Data":"3873af55b32c71f8f17b1b7e104a6b1c1ae67fba5a2444df389b5692bdb41f7c"}
Oct 30 21:46:07.157779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.157753    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:07.159413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.159388    6695 generic.go:356] "Generic (PLEG): container finished" podID="2190db3a-850e-4ca5-a71b-436879f0988a" containerID="2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213" exitCode=0
Oct 30 21:46:07.159545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.159500    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" event={"ID":"2190db3a-850e-4ca5-a71b-436879f0988a","Type":"ContainerDied","Data":"2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213"}
Oct 30 21:46:07.159545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.159538    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq" event={"ID":"2190db3a-850e-4ca5-a71b-436879f0988a","Type":"ContainerDied","Data":"27b975a09daa9ca334bf3f0a52b7900e8641c29a11d453fa66c184f8731987a6"}
Oct 30 21:46:07.165649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.160239    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"
Oct 30 21:46:07.165649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.163813    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy" pod "a711ee9f-5969-4295-92a6-c0e6a560a8e4" (UID: "a711ee9f-5969-4295-92a6-c0e6a560a8e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:07.165649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.164281    6695 generic.go:356] "Generic (PLEG): container finished" podID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerID="54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2" exitCode=0
Oct 30 21:46:07.165649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.164368    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" event={"ID":"a711ee9f-5969-4295-92a6-c0e6a560a8e4","Type":"ContainerDied","Data":"54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2"}
Oct 30 21:46:07.165649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.164424    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj" event={"ID":"a711ee9f-5969-4295-92a6-c0e6a560a8e4","Type":"ContainerDied","Data":"01a947e94eff46c1d2fb3f076912785a8908f01c89dbe17dc4cfd92263c77ede"}
Oct 30 21:46:07.165649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.164490    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"
Oct 30 21:46:07.170785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.170765    6695 scope.go:122] "RemoveContainer" containerID="ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8"
Oct 30 21:46:07.171101 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:07.171080    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8\": container with ID starting with ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8 not found: ID does not exist" containerID="ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8"
Oct 30 21:46:07.171207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.171107    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8"} err="failed to get container status \"ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8\": rpc error: code = NotFound desc = could not find container \"ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8\": container with ID starting with ea70cdade0e1ef2d5e760bd2a4a5c90122939cb68c9e5a02251ab0909432a0f8 not found: ID does not exist"
Oct 30 21:46:07.171207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.171124    6695 scope.go:122] "RemoveContainer" containerID="ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc"
Oct 30 21:46:07.184511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.184489    6695 scope.go:122] "RemoveContainer" containerID="ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc"
Oct 30 21:46:07.184865 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:07.184833    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc\": container with ID starting with ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc not found: ID does not exist" containerID="ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc"
Oct 30 21:46:07.184967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.184872    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc"} err="failed to get container status \"ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc\": rpc error: code = NotFound desc = could not find container \"ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc\": container with ID starting with ed0052293cd8a65c606fba11a777bd1a279c450bb16eac48236c8f74a15cf0dc not found: ID does not exist"
Oct 30 21:46:07.184967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.184890    6695 scope.go:122] "RemoveContainer" containerID="2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213"
Oct 30 21:46:07.195182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.195146    6695 scope.go:122] "RemoveContainer" containerID="2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213"
Oct 30 21:46:07.195507 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:07.195482    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213\": container with ID starting with 2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213 not found: ID does not exist" containerID="2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213"
Oct 30 21:46:07.195620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.195514    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213"} err="failed to get container status \"2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213\": rpc error: code = NotFound desc = could not find container \"2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213\": container with ID starting with 2c879e75c6b8b9b976a6920f705b75775da149501e8223d9f40d1acde5559213 not found: ID does not exist"
Oct 30 21:46:07.195620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.195533    6695 scope.go:122] "RemoveContainer" containerID="54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2"
Oct 30 21:46:07.206019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.206000    6695 scope.go:122] "RemoveContainer" containerID="54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2"
Oct 30 21:46:07.206357 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:07.206337    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2\": container with ID starting with 54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2 not found: ID does not exist" containerID="54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2"
Oct 30 21:46:07.206410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.206368    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2"} err="failed to get container status \"54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2\": rpc error: code = NotFound desc = could not find container \"54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2\": container with ID starting with 54420a328496eb2c05aa4a9317b065771f15222d71c56cf7e6f7a8d0cca8e9d2 not found: ID does not exist"
Oct 30 21:46:07.245407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.245374    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"]
Oct 30 21:46:07.252323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.252287    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-k58lq"]
Oct 30 21:46:07.256828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.256803    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqk6d\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-kube-api-access-vqk6d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.256828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.256827    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.257002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.256839    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.257002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.256857    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a711ee9f-5969-4295-92a6-c0e6a560a8e4-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:07.283858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.283827    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"]
Oct 30 21:46:07.287323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.287295    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-hn2lj"]
Oct 30 21:46:07.332642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.332585    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"]
Oct 30 21:46:07.337141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.337116    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-b8jrp"]
Oct 30 21:46:07.355447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.355408    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"]
Oct 30 21:46:07.363737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.363706    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-xjbhm"]
Oct 30 21:46:07.604863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:07.604818    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"]
Oct 30 21:46:09.247813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:09.247775    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" path="/var/lib/kubelet/pods/2190db3a-850e-4ca5-a71b-436879f0988a/volumes"
Oct 30 21:46:09.248344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:09.248327    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" path="/var/lib/kubelet/pods/a287432e-f634-4fbd-bb68-7e04f867f6c2/volumes"
Oct 30 21:46:09.248810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:09.248795    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" path="/var/lib/kubelet/pods/a711ee9f-5969-4295-92a6-c0e6a560a8e4/volumes"
Oct 30 21:46:09.249586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:09.249567    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" path="/var/lib/kubelet/pods/ea3a6a78-9914-46c0-b85b-6efb55452ade/volumes"
Oct 30 21:46:17.149583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:17.149547    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"]
Oct 30 21:46:20.694759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.694724    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"]
Oct 30 21:46:20.962815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.962730    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7403-46a563"]
Oct 30 21:46:20.963386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963371    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy"
Oct 30 21:46:20.963428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963388    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy"
Oct 30 21:46:20.963428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963408    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy"
Oct 30 21:46:20.963428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963414    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy"
Oct 30 21:46:20.963528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963439    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy"
Oct 30 21:46:20.963528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963445    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy"
Oct 30 21:46:20.963528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963469    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy"
Oct 30 21:46:20.963528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963474    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy"
Oct 30 21:46:20.963749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963574    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2190db3a-850e-4ca5-a71b-436879f0988a" containerName="istio-proxy"
Oct 30 21:46:20.963749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963587    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3a6a78-9914-46c0-b85b-6efb55452ade" containerName="istio-proxy"
Oct 30 21:46:20.963749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963598    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a711ee9f-5969-4295-92a6-c0e6a560a8e4" containerName="istio-proxy"
Oct 30 21:46:20.963749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.963619    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a287432e-f634-4fbd-bb68-7e04f867f6c2" containerName="istio-proxy"
Oct 30 21:46:20.967172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.967154    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:20.981054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.981026    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-46a563"]
Oct 30 21:46:20.983238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:20.983212    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vfm5w\""
Oct 30 21:46:21.089057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.089002    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm9m\" (UniqueName: \"kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m\") pod \"test-curl-7403-46a563\" (UID: \"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc\") " pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:21.190676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.190628    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm9m\" (UniqueName: \"kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m\") pod \"test-curl-7403-46a563\" (UID: \"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc\") " pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:21.209194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.209160    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm9m\" (UniqueName: \"kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m\") pod \"test-curl-7403-46a563\" (UID: \"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc\") " pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:21.280632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.280591    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:21.592899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.592815    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-46a563"]
Oct 30 21:46:21.596888 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:21.596851    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda92cb69_6c6e_4c8d_9cbf_fe742e8d24fc.slice/crio-5f459cc4966319d893cb40ffa8f0e277b9b672c97562c5bf51179aaa0f357b39 WatchSource:0}: Error finding container 5f459cc4966319d893cb40ffa8f0e277b9b672c97562c5bf51179aaa0f357b39: Status 404 returned error can't find the container with id 5f459cc4966319d893cb40ffa8f0e277b9b672c97562c5bf51179aaa0f357b39
Oct 30 21:46:21.654157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.654126    6695 scope.go:122] "RemoveContainer" containerID="cea834ce0c1617359a5163290c38a400315b6b244f56ee7c2b7a29d92ef4ba6c"
Oct 30 21:46:21.665503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.665481    6695 scope.go:122] "RemoveContainer" containerID="3f94f78eb588d0043f147a34b8600f1de93d2adc78890062dcce2348f969b0fe"
Oct 30 21:46:21.675902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.675873    6695 scope.go:122] "RemoveContainer" containerID="da85deb03831ce360548e6a007928771ec5cc3cda96e24b636600d83b3fca253"
Oct 30 21:46:21.688224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.688197    6695 scope.go:122] "RemoveContainer" containerID="2dd4de58d5fca76b9a84631be7fc90f2a09647dad384c7a2a4e99f4192395e84"
Oct 30 21:46:21.699170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.699147    6695 scope.go:122] "RemoveContainer" containerID="0f3dbaece1617d5a98b0935a23f2beb9927f1cf96cf944ea6cad4d967715d3b9"
Oct 30 21:46:21.709568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.709543    6695 scope.go:122] "RemoveContainer" containerID="03f4af3f2b2aec74d76f15155dc1f8a309da58f30db05bee2edfa340f21e9782"
Oct 30 21:46:21.719935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.719911    6695 scope.go:122] "RemoveContainer" containerID="15e5d22b397b9ae800a04dbaf8e2f8c9bf4df3c4f4f200f24c5d329c7b461578"
Oct 30 21:46:21.731960 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.731934    6695 scope.go:122] "RemoveContainer" containerID="c3d4bb68e8eef4d65026325cda6002276a25e814189b64a777265d6db6bbe39a"
Oct 30 21:46:21.742511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.742490    6695 scope.go:122] "RemoveContainer" containerID="ce158dfae0858d930cbacbc3d5bd6cec023138c5f4fb3ac99f78df043b956e1f"
Oct 30 21:46:21.756412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:21.756387    6695 scope.go:122] "RemoveContainer" containerID="0821ad68a07a0d446fa9484ff95f34a127cc823a334613b4fcdc4c1cb7dc9483"
Oct 30 21:46:22.319794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:22.319749    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-46a563" event={"ID":"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc","Type":"ContainerStarted","Data":"c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac"}
Oct 30 21:46:22.319794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:22.319801    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-46a563" event={"ID":"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc","Type":"ContainerStarted","Data":"5f459cc4966319d893cb40ffa8f0e277b9b672c97562c5bf51179aaa0f357b39"}
Oct 30 21:46:22.356924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:22.356858    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7403-46a563" podStartSLOduration=2.356832957 podStartE2EDuration="2.356832957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:22.353455227 +0000 UTC m=+2165.275913525" watchObservedRunningTime="2026-10-30 21:46:22.356832957 +0000 UTC m=+2165.279291256"
Oct 30 21:46:22.633762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:22.633643    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw" podUID="d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" containerName="main" containerID="cri-o://87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a" gracePeriod=45
Oct 30 21:46:22.928323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:22.928296    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:46:23.019858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.019824    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir\") pod \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") "
Oct 30 21:46:23.020072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.019884    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm\") pod \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") "
Oct 30 21:46:23.020072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.019924    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs\") pod \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") "
Oct 30 21:46:23.020072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c\" (UniqueName: \"kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c\") pod \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") "
Oct 30 21:46:23.020072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020058    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache\") pod \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") "
Oct 30 21:46:23.020391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020115    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir" pod "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" (UID: "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:23.020391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020154    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home\") pod \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\" (UID: \"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1\") "
Oct 30 21:46:23.020391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020311    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache" pod "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" (UID: "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:23.020559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020455    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home" pod "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" (UID: "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:23.020803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020786    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:23.020884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020805    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:23.020884 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.020814    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:23.022587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.022553    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs" pod "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" (UID: "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:23.022741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.022588    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c" pod "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" (UID: "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1"). InnerVolumeSpecName "kube-api-access-h827c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:23.022741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.022617    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm" pod "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" (UID: "d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:23.122075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.122040    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h827c\" (UniqueName: \"kubernetes.io/projected/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-kube-api-access-h827c\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:23.122075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.122074    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:23.122301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.122090    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:23.221246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.221130    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7403-46a563"]
Oct 30 21:46:23.226420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.226383    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7403-46a563"]
Oct 30 21:46:23.331717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.331672    6695 generic.go:356] "Generic (PLEG): container finished" podID="d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" containerID="87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a" exitCode=0
Oct 30 21:46:23.331920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.331729    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"
Oct 30 21:46:23.331920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.331747    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw" event={"ID":"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1","Type":"ContainerDied","Data":"87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a"}
Oct 30 21:46:23.331920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.331792    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw" event={"ID":"d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1","Type":"ContainerDied","Data":"327a81aeb22f8ec4ed153de8ace1fffd69b22807614653f3de4c38cdedb956f4"}
Oct 30 21:46:23.331920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.331811    6695 scope.go:122] "RemoveContainer" containerID="87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a"
Oct 30 21:46:23.342303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.342282    6695 scope.go:122] "RemoveContainer" containerID="87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a"
Oct 30 21:46:23.342625 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:23.342591    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a\": container with ID starting with 87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a not found: ID does not exist" containerID="87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a"
Oct 30 21:46:23.342741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.342628    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a"} err="failed to get container status \"87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a\": rpc error: code = NotFound desc = could not find container \"87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a\": container with ID starting with 87f2be31ec23ca7f57acfdc0db1418432dbf922c8ec47ee3cdce9b67fb8a920a not found: ID does not exist"
Oct 30 21:46:23.388438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.388378    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"]
Oct 30 21:46:23.418055 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.418016    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-0d74b3/noten-model-0d74b3-kserve-5fd8dfb757-vr2sw"]
Oct 30 21:46:23.451725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.451682    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7403-1a9dd9"]
Oct 30 21:46:23.452681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.452653    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" containerName="main"
Oct 30 21:46:23.452681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.452679    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" containerName="main"
Oct 30 21:46:23.452909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.452710    6695 cpu_manager.go:414] "containerMap: removing container" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" containerName="test-curl-7403-46a563"
Oct 30 21:46:23.452909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.452718    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" containerName="test-curl-7403-46a563"
Oct 30 21:46:23.452909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.452896    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" containerName="main"
Oct 30 21:46:23.452909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.452910    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" containerName="test-curl-7403-46a563"
Oct 30 21:46:23.458159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.458127    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-1a9dd9"]
Oct 30 21:46:23.458318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.458220    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:23.544953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.544919    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8jg\" (UniqueName: \"kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg\") pod \"test-curl-7403-1a9dd9\" (UID: \"fb6bf46a-7cb1-48e2-b137-07ed54b05438\") " pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:23.646257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.646221    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8jg\" (UniqueName: \"kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg\") pod \"test-curl-7403-1a9dd9\" (UID: \"fb6bf46a-7cb1-48e2-b137-07ed54b05438\") " pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:23.664768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.664732    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8jg\" (UniqueName: \"kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg\") pod \"test-curl-7403-1a9dd9\" (UID: \"fb6bf46a-7cb1-48e2-b137-07ed54b05438\") " pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:23.773620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:23.773564    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:24.084201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:24.084172    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-1a9dd9"]
Oct 30 21:46:24.087265 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:24.087232    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb6bf46a_7cb1_48e2_b137_07ed54b05438.slice/crio-42879a72962b7ec739e7dbfa41dbaabaef4eb45fa85f14623ed06aa781032b65 WatchSource:0}: Error finding container 42879a72962b7ec739e7dbfa41dbaabaef4eb45fa85f14623ed06aa781032b65: Status 404 returned error can't find the container with id 42879a72962b7ec739e7dbfa41dbaabaef4eb45fa85f14623ed06aa781032b65
Oct 30 21:46:24.341992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:24.341897    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-1a9dd9" event={"ID":"fb6bf46a-7cb1-48e2-b137-07ed54b05438","Type":"ContainerStarted","Data":"239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242"}
Oct 30 21:46:24.341992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:24.341939    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-1a9dd9" event={"ID":"fb6bf46a-7cb1-48e2-b137-07ed54b05438","Type":"ContainerStarted","Data":"42879a72962b7ec739e7dbfa41dbaabaef4eb45fa85f14623ed06aa781032b65"}
Oct 30 21:46:24.341992 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:24.341946    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7403-46a563" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" containerName="test-curl-7403-46a563" containerID="cri-o://c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac" gracePeriod=2
Oct 30 21:46:24.354730 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:24.354689    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-46a563\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:24.408478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:24.408426    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7403-1a9dd9" podStartSLOduration=1.408411454 podStartE2EDuration="1.408411454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:24.406982507 +0000 UTC m=+2167.329440807" watchObservedRunningTime="2026-10-30 21:46:24.408411454 +0000 UTC m=+2167.330869749"
Oct 30 21:46:24.421288 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:24.421245    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-46a563\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:25.224551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.224511    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1" path="/var/lib/kubelet/pods/d4ddea4c-22e3-4a2f-b1fc-9165b5ab85c1/volumes"
Oct 30 21:46:25.618736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.618700    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7403-1a9dd9"]
Oct 30 21:46:25.628224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.628120    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7403-1a9dd9"]
Oct 30 21:46:25.776994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.776955    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7403-d24a11"]
Oct 30 21:46:25.777588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.777573    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" containerName="test-curl-7403-1a9dd9"
Oct 30 21:46:25.777658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.777590    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" containerName="test-curl-7403-1a9dd9"
Oct 30 21:46:25.777748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.777738    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" containerName="test-curl-7403-1a9dd9"
Oct 30 21:46:25.781353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.781335    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:25.798012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.797986    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-d24a11"]
Oct 30 21:46:25.918529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:25.918439    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjf7\" (UniqueName: \"kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7\") pod \"test-curl-7403-d24a11\" (UID: \"94e03fab-7034-4805-b9f9-bc8893bdf77f\") " pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:26.020004 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.019965    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjf7\" (UniqueName: \"kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7\") pod \"test-curl-7403-d24a11\" (UID: \"94e03fab-7034-4805-b9f9-bc8893bdf77f\") " pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:26.038179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.038151    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjf7\" (UniqueName: \"kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7\") pod \"test-curl-7403-d24a11\" (UID: \"94e03fab-7034-4805-b9f9-bc8893bdf77f\") " pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:26.094349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.094310    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:26.364537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.364502    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7403-1a9dd9" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" containerName="test-curl-7403-1a9dd9" containerID="cri-o://239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242" gracePeriod=2
Oct 30 21:46:26.377092 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:26.377048    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-1a9dd9\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:26.397740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.397714    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-d24a11"]
Oct 30 21:46:26.414750 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:26.414715    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e03fab_7034_4805_b9f9_bc8893bdf77f.slice/crio-133df3cc7594e6ee0d70aa089555122b2ce2339614f3afd3837582417e4477aa WatchSource:0}: Error finding container 133df3cc7594e6ee0d70aa089555122b2ce2339614f3afd3837582417e4477aa: Status 404 returned error can't find the container with id 133df3cc7594e6ee0d70aa089555122b2ce2339614f3afd3837582417e4477aa
Oct 30 21:46:26.541192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.541164    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:26.553799 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:26.553766    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-1a9dd9\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:26.629902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.629806    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m\" (UniqueName: \"kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m\") pod \"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc\" (UID: \"da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc\") "
Oct 30 21:46:26.631922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.631895    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m" pod "da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" (UID: "da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc"). InnerVolumeSpecName "kube-api-access-lmm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:26.731845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:26.731807    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmm9m\" (UniqueName: \"kubernetes.io/projected/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc-kube-api-access-lmm9m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:27.215155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.215120    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" path="/var/lib/kubelet/pods/da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc/volumes"
Oct 30 21:46:27.223185 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:27.223153    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-1a9dd9\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:27.374086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.373992    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-d24a11" event={"ID":"94e03fab-7034-4805-b9f9-bc8893bdf77f","Type":"ContainerStarted","Data":"705fd28c41f73bda7c90847aa48e7b9948df214d221a8dd7768dcae9bd1730b4"}
Oct 30 21:46:27.374086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.374048    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-d24a11" event={"ID":"94e03fab-7034-4805-b9f9-bc8893bdf77f","Type":"ContainerStarted","Data":"133df3cc7594e6ee0d70aa089555122b2ce2339614f3afd3837582417e4477aa"}
Oct 30 21:46:27.375753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.375725    6695 generic.go:356] "Generic (PLEG): container finished" podID="da92cb69-6c6e-4c8d-9cbf-fe742e8d24fc" containerID="c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac" exitCode=137
Oct 30 21:46:27.375889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.375763    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-46a563"
Oct 30 21:46:27.375889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.375781    6695 scope.go:122] "RemoveContainer" containerID="c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac"
Oct 30 21:46:27.387068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.387046    6695 scope.go:122] "RemoveContainer" containerID="c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac"
Oct 30 21:46:27.387427 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:27.387401    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac\": container with ID starting with c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac not found: ID does not exist" containerID="c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac"
Oct 30 21:46:27.387579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.387434    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac"} err="failed to get container status \"c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac\": rpc error: code = NotFound desc = could not find container \"c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac\": container with ID starting with c6952839e8dbcf63c765e300cad2bfca8abe96be9f2a661c7598d1d5036947ac not found: ID does not exist"
Oct 30 21:46:27.412311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.412277    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"]
Oct 30 21:46:27.413046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.413001    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7403-d24a11" podStartSLOduration=2.412988638 podStartE2EDuration="2.412988638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:27.411889424 +0000 UTC m=+2170.334347721" watchObservedRunningTime="2026-10-30 21:46:27.412988638 +0000 UTC m=+2170.335446937"
Oct 30 21:46:27.418277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.418250    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.432348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.432317    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:27.432348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.432329    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:27.432573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.432404    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:46:27.436033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.436002    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"]
Oct 30 21:46:27.443041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.443018    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:27.542773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.542734    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qbr7\" (UniqueName: \"kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.542967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.542803    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.542967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.542893    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.623447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.623415    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"]
Oct 30 21:46:27.628118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.628061    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.644236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.644192    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qbr7\" (UniqueName: \"kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.644394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.644251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.644394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.644279    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.645168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.645140    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.647393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.647366    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.649353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.649328    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"]
Oct 30 21:46:27.664798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.664763    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qbr7\" (UniqueName: \"kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7\") pod \"maas-api-revoke-sub-8c960f59-q7b29\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.733447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.733412    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:27.745402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.745373    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkd8\" (UniqueName: \"kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.745558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.745457    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.745558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.745543    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.846721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.846685    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkd8\" (UniqueName: \"kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.846901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.846789    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.846901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.846856    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.847691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.847666    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.850255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.850207    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.866277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.866243    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkd8\" (UniqueName: \"kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8\") pod \"maas-api-revoke-sub-386fb9d7-r672p\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:27.941994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:27.941904    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:28.042576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.042540    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"]
Oct 30 21:46:28.045175 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:28.045140    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088b97ad_b094_4cc3_bd7d_04ae74d25c23.slice/crio-70ddc8ef3fe0d161100a2687e94793a473cf1420c8b2a14c697e4f1c2039e6d5 WatchSource:0}: Error finding container 70ddc8ef3fe0d161100a2687e94793a473cf1420c8b2a14c697e4f1c2039e6d5: Status 404 returned error can't find the container with id 70ddc8ef3fe0d161100a2687e94793a473cf1420c8b2a14c697e4f1c2039e6d5
Oct 30 21:46:28.143156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.143117    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"]
Oct 30 21:46:28.143355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.143328    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy" containerID="cri-o://0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8" gracePeriod=30
Oct 30 21:46:28.263780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.263743    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"]
Oct 30 21:46:28.265936 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:28.265905    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fb5bda_416c_4d9f_87ff_b8b478b5672b.slice/crio-2463bf7266ce54591c353956d04532df19d2fc9d31f5df5fe4920f5699580d60 WatchSource:0}: Error finding container 2463bf7266ce54591c353956d04532df19d2fc9d31f5df5fe4920f5699580d60: Status 404 returned error can't find the container with id 2463bf7266ce54591c353956d04532df19d2fc9d31f5df5fe4920f5699580d60
Oct 30 21:46:28.289176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.289151    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7403-d24a11"]
Oct 30 21:46:28.293990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.293961    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7403-d24a11"]
Oct 30 21:46:28.385417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.385365    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p" event={"ID":"c6fb5bda-416c-4d9f-87ff-b8b478b5672b","Type":"ContainerStarted","Data":"2463bf7266ce54591c353956d04532df19d2fc9d31f5df5fe4920f5699580d60"}
Oct 30 21:46:28.386923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.386895    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29" event={"ID":"088b97ad-b094-4cc3-bd7d-04ae74d25c23","Type":"ContainerStarted","Data":"70ddc8ef3fe0d161100a2687e94793a473cf1420c8b2a14c697e4f1c2039e6d5"}
Oct 30 21:46:28.436437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.436406    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7403-3b982d"]
Oct 30 21:46:28.437102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.437076    6695 cpu_manager.go:414] "containerMap: removing container" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" containerName="test-curl-7403-d24a11"
Oct 30 21:46:28.437102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.437102    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" containerName="test-curl-7403-d24a11"
Oct 30 21:46:28.437243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.437229    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" containerName="test-curl-7403-d24a11"
Oct 30 21:46:28.442475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.442453    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:28.457969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.457925    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-3b982d"]
Oct 30 21:46:28.563114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.563073    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzckt\" (UniqueName: \"kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt\") pod \"test-curl-7403-3b982d\" (UID: \"96ae21a7-04d8-4b68-95cd-5d7174ec84ae\") " pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:28.596974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.596948    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:28.664017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.663982    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg\" (UniqueName: \"kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg\") pod \"fb6bf46a-7cb1-48e2-b137-07ed54b05438\" (UID: \"fb6bf46a-7cb1-48e2-b137-07ed54b05438\") "
Oct 30 21:46:28.664422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.664405    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzckt\" (UniqueName: \"kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt\") pod \"test-curl-7403-3b982d\" (UID: \"96ae21a7-04d8-4b68-95cd-5d7174ec84ae\") " pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:28.666090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.666067    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg" pod "fb6bf46a-7cb1-48e2-b137-07ed54b05438" (UID: "fb6bf46a-7cb1-48e2-b137-07ed54b05438"). InnerVolumeSpecName "kube-api-access-hr8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:28.689100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.689072    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzckt\" (UniqueName: \"kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt\") pod \"test-curl-7403-3b982d\" (UID: \"96ae21a7-04d8-4b68-95cd-5d7174ec84ae\") " pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:28.766120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.766034    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8jg\" (UniqueName: \"kubernetes.io/projected/fb6bf46a-7cb1-48e2-b137-07ed54b05438-kube-api-access-hr8jg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:28.792377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:28.792317    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:29.143826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.143559    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-3b982d"]
Oct 30 21:46:29.145839 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:29.145798    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96ae21a7_04d8_4b68_95cd_5d7174ec84ae.slice/crio-0d12a08f4dcdebaad39986df26d90e7cd82755417175cb0649defaf6e85dc688 WatchSource:0}: Error finding container 0d12a08f4dcdebaad39986df26d90e7cd82755417175cb0649defaf6e85dc688: Status 404 returned error can't find the container with id 0d12a08f4dcdebaad39986df26d90e7cd82755417175cb0649defaf6e85dc688
Oct 30 21:46:29.219364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.219330    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" path="/var/lib/kubelet/pods/fb6bf46a-7cb1-48e2-b137-07ed54b05438/volumes"
Oct 30 21:46:29.396304 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.396208    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-3b982d" event={"ID":"96ae21a7-04d8-4b68-95cd-5d7174ec84ae","Type":"ContainerStarted","Data":"2964c0e17e26c172ee3e0158f57206ac2aea62fd157ed8e57afd035548c7bcd5"}
Oct 30 21:46:29.396780 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.396311    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-3b982d" event={"ID":"96ae21a7-04d8-4b68-95cd-5d7174ec84ae","Type":"ContainerStarted","Data":"0d12a08f4dcdebaad39986df26d90e7cd82755417175cb0649defaf6e85dc688"}
Oct 30 21:46:29.397916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.397889    6695 generic.go:356] "Generic (PLEG): container finished" podID="fb6bf46a-7cb1-48e2-b137-07ed54b05438" containerID="239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242" exitCode=137
Oct 30 21:46:29.398044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.397932    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-1a9dd9"
Oct 30 21:46:29.398044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.397989    6695 scope.go:122] "RemoveContainer" containerID="239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242"
Oct 30 21:46:29.399827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.399802    6695 generic.go:356] "Generic (PLEG): container finished" podID="c6fb5bda-416c-4d9f-87ff-b8b478b5672b" containerID="a4bd05406baf0e46e6adee1e7109cb02f5fdada9d361dc7067030066f382b797" exitCode=0
Oct 30 21:46:29.400322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.400293    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p" event={"ID":"c6fb5bda-416c-4d9f-87ff-b8b478b5672b","Type":"ContainerDied","Data":"a4bd05406baf0e46e6adee1e7109cb02f5fdada9d361dc7067030066f382b797"}
Oct 30 21:46:29.401957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.401932    6695 generic.go:356] "Generic (PLEG): container finished" podID="088b97ad-b094-4cc3-bd7d-04ae74d25c23" containerID="cb15c9cb87f2ef896e8502880ac61605c7e9b28bad5aca014487a7f3e5cad14e" exitCode=0
Oct 30 21:46:29.402063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.402010    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29" event={"ID":"088b97ad-b094-4cc3-bd7d-04ae74d25c23","Type":"ContainerDied","Data":"cb15c9cb87f2ef896e8502880ac61605c7e9b28bad5aca014487a7f3e5cad14e"}
Oct 30 21:46:29.402141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.402099    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7403-d24a11" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" containerName="test-curl-7403-d24a11" containerID="cri-o://705fd28c41f73bda7c90847aa48e7b9948df214d221a8dd7768dcae9bd1730b4" gracePeriod=2
Oct 30 21:46:29.408534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.408514    6695 scope.go:122] "RemoveContainer" containerID="239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242"
Oct 30 21:46:29.408913 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:29.408891    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242\": container with ID starting with 239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242 not found: ID does not exist" containerID="239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242"
Oct 30 21:46:29.408990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.408919    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242"} err="failed to get container status \"239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242\": rpc error: code = NotFound desc = could not find container \"239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242\": container with ID starting with 239aa897a3d4da3905c63abb07b793143c1762c81513bd95fa5e766e72826242 not found: ID does not exist"
Oct 30 21:46:29.433920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:29.433876    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7403-3b982d" podStartSLOduration=1.43386335 podStartE2EDuration="1.43386335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:29.430719908 +0000 UTC m=+2172.353178205" watchObservedRunningTime="2026-10-30 21:46:29.43386335 +0000 UTC m=+2172.356321647"
Oct 30 21:46:30.249593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.249549    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7403-3b982d"]
Oct 30 21:46:30.256478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.256441    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7403-3b982d"]
Oct 30 21:46:30.401694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.401656    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7403-024fac"]
Oct 30 21:46:30.406367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.406327    6695 cpu_manager.go:414] "containerMap: removing container" podUID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" containerName="test-curl-7403-3b982d"
Oct 30 21:46:30.406367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.406364    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" containerName="test-curl-7403-3b982d"
Oct 30 21:46:30.406728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.406710    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" containerName="test-curl-7403-3b982d"
Oct 30 21:46:30.413339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.413305    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:30.427268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.427233    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-024fac"]
Oct 30 21:46:30.450227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.449737    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffmd\" (UniqueName: \"kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd\") pod \"test-curl-7403-024fac\" (UID: \"702ee375-291c-4ed3-b5d1-981b14d795c4\") " pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:30.561693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.561034    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffmd\" (UniqueName: \"kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd\") pod \"test-curl-7403-024fac\" (UID: \"702ee375-291c-4ed3-b5d1-981b14d795c4\") " pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:30.591267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.590175    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffmd\" (UniqueName: \"kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd\") pod \"test-curl-7403-024fac\" (UID: \"702ee375-291c-4ed3-b5d1-981b14d795c4\") " pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:30.662575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.662544    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:30.721305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.721265    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:30.739296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.739260    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:30.765415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.763858    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7\" (UniqueName: \"kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7\") pod \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") "
Oct 30 21:46:30.765415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.763940    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca\") pod \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") "
Oct 30 21:46:30.765415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.763988    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca\") pod \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") "
Oct 30 21:46:30.765415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.764215    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8\" (UniqueName: \"kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8\") pod \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") "
Oct 30 21:46:30.765415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.764263    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca\") pod \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\" (UID: \"088b97ad-b094-4cc3-bd7d-04ae74d25c23\") "
Oct 30 21:46:30.765415 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.764343    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca\") pod \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\" (UID: \"c6fb5bda-416c-4d9f-87ff-b8b478b5672b\") "
Oct 30 21:46:30.766673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.766427    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca" pod "c6fb5bda-416c-4d9f-87ff-b8b478b5672b" (UID: "c6fb5bda-416c-4d9f-87ff-b8b478b5672b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:30.766976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.766940    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca" pod "088b97ad-b094-4cc3-bd7d-04ae74d25c23" (UID: "088b97ad-b094-4cc3-bd7d-04ae74d25c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:30.771905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.771865    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7" pod "088b97ad-b094-4cc3-bd7d-04ae74d25c23" (UID: "088b97ad-b094-4cc3-bd7d-04ae74d25c23"). InnerVolumeSpecName "kube-api-access-2qbr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:30.772489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.772451    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca" pod "088b97ad-b094-4cc3-bd7d-04ae74d25c23" (UID: "088b97ad-b094-4cc3-bd7d-04ae74d25c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:30.773521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.773499    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca" pod "c6fb5bda-416c-4d9f-87ff-b8b478b5672b" (UID: "c6fb5bda-416c-4d9f-87ff-b8b478b5672b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:30.775468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.775369    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8" pod "c6fb5bda-416c-4d9f-87ff-b8b478b5672b" (UID: "c6fb5bda-416c-4d9f-87ff-b8b478b5672b"). InnerVolumeSpecName "kube-api-access-wlkd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:30.880429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.880393    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:30.880912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.880880    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkd8\" (UniqueName: \"kubernetes.io/projected/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-kube-api-access-wlkd8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:30.881109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.881094    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:30.881214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.881202    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fb5bda-416c-4d9f-87ff-b8b478b5672b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:30.881315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.881304    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qbr7\" (UniqueName: \"kubernetes.io/projected/088b97ad-b094-4cc3-bd7d-04ae74d25c23-kube-api-access-2qbr7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:30.881401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:30.881390    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/088b97ad-b094-4cc3-bd7d-04ae74d25c23-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:31.061231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.061145    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7403-024fac"]
Oct 30 21:46:31.065035 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:31.065000    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod702ee375_291c_4ed3_b5d1_981b14d795c4.slice/crio-26a196bd34b880046691396e073b0280a8bae677dcc4e38b10bfd51356a7da8c WatchSource:0}: Error finding container 26a196bd34b880046691396e073b0280a8bae677dcc4e38b10bfd51356a7da8c: Status 404 returned error can't find the container with id 26a196bd34b880046691396e073b0280a8bae677dcc4e38b10bfd51356a7da8c
Oct 30 21:46:31.081430 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:31.081393    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:46:31.458095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.458066    6695 generic.go:356] "Generic (PLEG): container finished" podID="94e03fab-7034-4805-b9f9-bc8893bdf77f" containerID="705fd28c41f73bda7c90847aa48e7b9948df214d221a8dd7768dcae9bd1730b4" exitCode=137
Oct 30 21:46:31.460049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.459955    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"
Oct 30 21:46:31.460049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.459966    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29" event={"ID":"088b97ad-b094-4cc3-bd7d-04ae74d25c23","Type":"ContainerDied","Data":"70ddc8ef3fe0d161100a2687e94793a473cf1420c8b2a14c697e4f1c2039e6d5"}
Oct 30 21:46:31.460049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.460024    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70ddc8ef3fe0d161100a2687e94793a473cf1420c8b2a14c697e4f1c2039e6d5"
Oct 30 21:46:31.462253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.462202    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-024fac" event={"ID":"702ee375-291c-4ed3-b5d1-981b14d795c4","Type":"ContainerStarted","Data":"ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76"}
Oct 30 21:46:31.462253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.462233    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7403-024fac" event={"ID":"702ee375-291c-4ed3-b5d1-981b14d795c4","Type":"ContainerStarted","Data":"26a196bd34b880046691396e073b0280a8bae677dcc4e38b10bfd51356a7da8c"}
Oct 30 21:46:31.464868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.464840    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"
Oct 30 21:46:31.464984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.464870    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p" event={"ID":"c6fb5bda-416c-4d9f-87ff-b8b478b5672b","Type":"ContainerDied","Data":"2463bf7266ce54591c353956d04532df19d2fc9d31f5df5fe4920f5699580d60"}
Oct 30 21:46:31.464984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.464929    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2463bf7266ce54591c353956d04532df19d2fc9d31f5df5fe4920f5699580d60"
Oct 30 21:46:31.465097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.465081    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7403-3b982d" podUID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" containerName="test-curl-7403-3b982d" containerID="cri-o://2964c0e17e26c172ee3e0158f57206ac2aea62fd157ed8e57afd035548c7bcd5" gracePeriod=2
Oct 30 21:46:31.499433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.499377    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7403-024fac" podStartSLOduration=1.499362195 podStartE2EDuration="1.499362195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:31.497520176 +0000 UTC m=+2174.419978474" watchObservedRunningTime="2026-10-30 21:46:31.499362195 +0000 UTC m=+2174.421820493"
Oct 30 21:46:31.642657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.642632    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:31.695053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.694956    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7\" (UniqueName: \"kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7\") pod \"94e03fab-7034-4805-b9f9-bc8893bdf77f\" (UID: \"94e03fab-7034-4805-b9f9-bc8893bdf77f\") "
Oct 30 21:46:31.697903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.697865    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7" pod "94e03fab-7034-4805-b9f9-bc8893bdf77f" (UID: "94e03fab-7034-4805-b9f9-bc8893bdf77f"). InnerVolumeSpecName "kube-api-access-mpjf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:31.806195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:31.805979    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjf7\" (UniqueName: \"kubernetes.io/projected/94e03fab-7034-4805-b9f9-bc8893bdf77f-kube-api-access-mpjf7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.172565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.172524    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982" podUID="fee473a7-83af-4f47-bb06-fa7a9037e843" containerName="main" containerID="cri-o://8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533" gracePeriod=45
Oct 30 21:46:32.407788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.407756    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7403-024fac"]
Oct 30 21:46:32.414182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.414151    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7403-024fac"]
Oct 30 21:46:32.450593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.450558    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:46:32.474094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.473805    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:32.474094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.473830    6695 scope.go:122] "RemoveContainer" containerID="705fd28c41f73bda7c90847aa48e7b9948df214d221a8dd7768dcae9bd1730b4"
Oct 30 21:46:32.475468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.475439    6695 generic.go:356] "Generic (PLEG): container finished" podID="fee473a7-83af-4f47-bb06-fa7a9037e843" containerID="8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533" exitCode=0
Oct 30 21:46:32.475587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.475478    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"
Oct 30 21:46:32.475587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.475521    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982" event={"ID":"fee473a7-83af-4f47-bb06-fa7a9037e843","Type":"ContainerDied","Data":"8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533"}
Oct 30 21:46:32.475587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.475561    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982" event={"ID":"fee473a7-83af-4f47-bb06-fa7a9037e843","Type":"ContainerDied","Data":"8003000ac3f1cd54a9abb032967d925d6ce845fd747ea00dc708953df2f5eefa"}
Oct 30 21:46:32.484987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.484970    6695 scope.go:122] "RemoveContainer" containerID="8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533"
Oct 30 21:46:32.495107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.495084    6695 scope.go:122] "RemoveContainer" containerID="8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533"
Oct 30 21:46:32.495667 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:32.495633    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533\": container with ID starting with 8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533 not found: ID does not exist" containerID="8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533"
Oct 30 21:46:32.495814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.495677    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533"} err="failed to get container status \"8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533\": rpc error: code = NotFound desc = could not find container \"8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533\": container with ID starting with 8fc5f32ee56f8f41496113282f26132bc41c5cfbcc1efb2d2417067975fbd533 not found: ID does not exist"
Oct 30 21:46:32.506270 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:32.506233    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-d24a11\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:32.515707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.515682    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt\" (UniqueName: \"kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt\") pod \"fee473a7-83af-4f47-bb06-fa7a9037e843\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") "
Oct 30 21:46:32.515815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.515733    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home\") pod \"fee473a7-83af-4f47-bb06-fa7a9037e843\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") "
Oct 30 21:46:32.515876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.515815    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs\" (UniqueName: \"kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs\") pod \"fee473a7-83af-4f47-bb06-fa7a9037e843\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") "
Oct 30 21:46:32.515876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.515856    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm\") pod \"fee473a7-83af-4f47-bb06-fa7a9037e843\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") "
Oct 30 21:46:32.515998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.515929    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache\") pod \"fee473a7-83af-4f47-bb06-fa7a9037e843\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") "
Oct 30 21:46:32.516085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516060    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home" pod "fee473a7-83af-4f47-bb06-fa7a9037e843" (UID: "fee473a7-83af-4f47-bb06-fa7a9037e843"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:32.516134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516104    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir\") pod \"fee473a7-83af-4f47-bb06-fa7a9037e843\" (UID: \"fee473a7-83af-4f47-bb06-fa7a9037e843\") "
Oct 30 21:46:32.516282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache" pod "fee473a7-83af-4f47-bb06-fa7a9037e843" (UID: "fee473a7-83af-4f47-bb06-fa7a9037e843"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:32.516452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516420    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir" pod "fee473a7-83af-4f47-bb06-fa7a9037e843" (UID: "fee473a7-83af-4f47-bb06-fa7a9037e843"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:32.516870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516830    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.516870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516865    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.517042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.516881    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.518377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.518355    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt" pod "fee473a7-83af-4f47-bb06-fa7a9037e843" (UID: "fee473a7-83af-4f47-bb06-fa7a9037e843"). InnerVolumeSpecName "kube-api-access-wrqzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:32.518487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.518464    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs" pod "fee473a7-83af-4f47-bb06-fa7a9037e843" (UID: "fee473a7-83af-4f47-bb06-fa7a9037e843"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:32.518560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.518510    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm" pod "fee473a7-83af-4f47-bb06-fa7a9037e843" (UID: "fee473a7-83af-4f47-bb06-fa7a9037e843"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:32.618757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.618731    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqzt\" (UniqueName: \"kubernetes.io/projected/fee473a7-83af-4f47-bb06-fa7a9037e843-kube-api-access-wrqzt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.618757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.618755    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee473a7-83af-4f47-bb06-fa7a9037e843-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.618953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.618767    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee473a7-83af-4f47-bb06-fa7a9037e843-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:32.824799 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.824763    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"]
Oct 30 21:46:32.833002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:32.832976    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-09eabe/auth-test-model-09eabe-kserve-55bf7d78cc-pn982"]
Oct 30 21:46:32.842198 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:32.842164    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-d24a11\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" pod="opendatahub/test-curl-7403-d24a11"
Oct 30 21:46:33.215099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.215069    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e03fab-7034-4805-b9f9-bc8893bdf77f" path="/var/lib/kubelet/pods/94e03fab-7034-4805-b9f9-bc8893bdf77f/volumes"
Oct 30 21:46:33.215569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.215555    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee473a7-83af-4f47-bb06-fa7a9037e843" path="/var/lib/kubelet/pods/fee473a7-83af-4f47-bb06-fa7a9037e843/volumes"
Oct 30 21:46:33.425710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.425642    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:46:33.487774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.487598    6695 generic.go:356] "Generic (PLEG): container finished" podID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerID="0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8" exitCode=0
Oct 30 21:46:33.487774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.487650    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" event={"ID":"0e8aa9e3-4606-4ec8-acf2-7e044a590f72","Type":"ContainerDied","Data":"0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8"}
Oct 30 21:46:33.487774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.487692    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"
Oct 30 21:46:33.487774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.487712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng" event={"ID":"0e8aa9e3-4606-4ec8-acf2-7e044a590f72","Type":"ContainerDied","Data":"820c57d5f82dd2e29db386f43afc69c66752074d4fed98fa3959c45e26e9955a"}
Oct 30 21:46:33.487774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.487741    6695 scope.go:122] "RemoveContainer" containerID="0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8"
Oct 30 21:46:33.493698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.493667    6695 generic.go:356] "Generic (PLEG): container finished" podID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" containerID="2964c0e17e26c172ee3e0158f57206ac2aea62fd157ed8e57afd035548c7bcd5" exitCode=137
Oct 30 21:46:33.493947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.493925    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7403-024fac" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" containerName="test-curl-7403-024fac" containerID="cri-o://ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76" gracePeriod=2
Oct 30 21:46:33.524126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.524097    6695 scope.go:122] "RemoveContainer" containerID="0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8"
Oct 30 21:46:33.524536 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:33.524507    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8\": container with ID starting with 0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8 not found: ID does not exist" containerID="0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8"
Oct 30 21:46:33.524674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.524543    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8"} err="failed to get container status \"0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8\": rpc error: code = NotFound desc = could not find container \"0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8\": container with ID starting with 0f896aba5ebc9a082a1104d71ab69e174cb0faeb7acc387a5d68d3eb7ed4d8e8 not found: ID does not exist"
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.531259    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.531346    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.531482    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.531515    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.531959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532036    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532045    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532178    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532292    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532354    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532394    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532477    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy\") pod \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\" (UID: \"0e8aa9e3-4606-4ec8-acf2-7e044a590f72\") "
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.532598    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.533138    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.533160    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.533174    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.533187    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.535642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.533197    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:33.538114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.538078    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:33.538354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.538333    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:33.538517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.538499    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "kube-api-access-lj64z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:33.538671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.538576    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy" pod "0e8aa9e3-4606-4ec8-acf2-7e044a590f72" (UID: "0e8aa9e3-4606-4ec8-acf2-7e044a590f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:33.640674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.638025    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.640674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.638069    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj64z\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-kube-api-access-lj64z\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.640674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.638085    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.640674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.638099    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.640674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.638114    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8aa9e3-4606-4ec8-acf2-7e044a590f72-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:33.710907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.710880    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:33.837954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.837915    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"]
Oct 30 21:46:33.840819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.840787    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt\" (UniqueName: \"kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt\") pod \"96ae21a7-04d8-4b68-95cd-5d7174ec84ae\" (UID: \"96ae21a7-04d8-4b68-95cd-5d7174ec84ae\") "
Oct 30 21:46:33.843058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.843023    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt" pod "96ae21a7-04d8-4b68-95cd-5d7174ec84ae" (UID: "96ae21a7-04d8-4b68-95cd-5d7174ec84ae"). InnerVolumeSpecName "kube-api-access-mzckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:33.847743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.847718    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-thfng"]
Oct 30 21:46:33.943377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:33.943279    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzckt\" (UniqueName: \"kubernetes.io/projected/96ae21a7-04d8-4b68-95cd-5d7174ec84ae-kube-api-access-mzckt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:34.454888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.454846    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"]
Oct 30 21:46:34.455662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455633    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c6fb5bda-416c-4d9f-87ff-b8b478b5672b" containerName="revoke-keys"
Oct 30 21:46:34.455662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455657    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fb5bda-416c-4d9f-87ff-b8b478b5672b" containerName="revoke-keys"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455679    6695 cpu_manager.go:414] "containerMap: removing container" podUID="088b97ad-b094-4cc3-bd7d-04ae74d25c23" containerName="revoke-keys"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455685    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088b97ad-b094-4cc3-bd7d-04ae74d25c23" containerName="revoke-keys"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455711    6695 cpu_manager.go:414] "containerMap: removing container" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" containerName="test-curl-7403-024fac"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455718    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" containerName="test-curl-7403-024fac"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455729    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455734    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455745    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fee473a7-83af-4f47-bb06-fa7a9037e843" containerName="main"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455750    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee473a7-83af-4f47-bb06-fa7a9037e843" containerName="main"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455847    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee473a7-83af-4f47-bb06-fa7a9037e843" containerName="main"
Oct 30 21:46:34.455863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455861    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fb5bda-416c-4d9f-87ff-b8b478b5672b" containerName="revoke-keys"
Oct 30 21:46:34.456421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455874    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="088b97ad-b094-4cc3-bd7d-04ae74d25c23" containerName="revoke-keys"
Oct 30 21:46:34.456421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455887    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" containerName="test-curl-7403-024fac"
Oct 30 21:46:34.456421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.455896    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" containerName="istio-proxy"
Oct 30 21:46:34.459820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.459775    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.474932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.474903    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c19461-openshift-default-dockercfg-dwvdn\""
Oct 30 21:46:34.482632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.482563    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"]
Oct 30 21:46:34.512752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.512719    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:34.513238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.512763    6695 scope.go:122] "RemoveContainer" containerID="2964c0e17e26c172ee3e0158f57206ac2aea62fd157ed8e57afd035548c7bcd5"
Oct 30 21:46:34.536431 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:34.536397    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-3b982d\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" pod="opendatahub/test-curl-7403-3b982d"
Oct 30 21:46:34.553938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.551582    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.554279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.554228    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.554551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.554528    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.554839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.554820    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pnj\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.555078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.555061    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.555299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.555267    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.555741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.555684    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.555866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.555755    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.555866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.555787    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665109    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665176    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pnj\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665230    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665276    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665396    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665437    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665460    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665578    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.666005    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.666226    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.666235    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.666695    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.667119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.666940    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.668097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.665620    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.711500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.711382    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pnj\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.714347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.713517    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.714347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.713813    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.716386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.716337    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token\") pod \"e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") " pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:34.779807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:34.779775    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:35.103379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.103262    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"]
Oct 30 21:46:35.106677 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:35.106591    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b41768_b63b_44df_94a1_292629eb7d29.slice/crio-ae13773d75b645a18c0c2bee48e627a4dead7e191c2428794afe4facfdbf2ebd WatchSource:0}: Error finding container ae13773d75b645a18c0c2bee48e627a4dead7e191c2428794afe4facfdbf2ebd: Status 404 returned error can't find the container with id ae13773d75b645a18c0c2bee48e627a4dead7e191c2428794afe4facfdbf2ebd
Oct 30 21:46:35.110201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.110162    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:35.110325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.110245    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:35.110325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.110284    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:46:35.215915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.215878    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8aa9e3-4606-4ec8-acf2-7e044a590f72" path="/var/lib/kubelet/pods/0e8aa9e3-4606-4ec8-acf2-7e044a590f72/volumes"
Oct 30 21:46:35.216440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.216412    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ae21a7-04d8-4b68-95cd-5d7174ec84ae" path="/var/lib/kubelet/pods/96ae21a7-04d8-4b68-95cd-5d7174ec84ae/volumes"
Oct 30 21:46:35.524538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.524482    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" event={"ID":"d0b41768-b63b-44df-94a1-292629eb7d29","Type":"ContainerStarted","Data":"7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"}
Oct 30 21:46:35.525062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.524553    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" event={"ID":"d0b41768-b63b-44df-94a1-292629eb7d29","Type":"ContainerStarted","Data":"ae13773d75b645a18c0c2bee48e627a4dead7e191c2428794afe4facfdbf2ebd"}
Oct 30 21:46:35.565877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.565824    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podStartSLOduration=1.565805984 podStartE2EDuration="1.565805984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:35.564233197 +0000 UTC m=+2178.486691495" watchObservedRunningTime="2026-10-30 21:46:35.565805984 +0000 UTC m=+2178.488264285"
Oct 30 21:46:35.728631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.726809    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" podUID="2d0be272-d582-4a67-8956-c820a5ed5ed4" containerName="main" containerID="cri-o://42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb" gracePeriod=45
Oct 30 21:46:35.780310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.780275    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:35.829143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.829116    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:35.850105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.850070    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" podUID="2d0be272-d582-4a67-8956-c820a5ed5ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.1.17:8000/ready\": dial tcp 10.134.1.17:8000: connect: connection refused"
Oct 30 21:46:35.906087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.906052    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd\" (UniqueName: \"kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd\") pod \"702ee375-291c-4ed3-b5d1-981b14d795c4\" (UID: \"702ee375-291c-4ed3-b5d1-981b14d795c4\") "
Oct 30 21:46:35.908276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:35.908244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd" pod "702ee375-291c-4ed3-b5d1-981b14d795c4" (UID: "702ee375-291c-4ed3-b5d1-981b14d795c4"). InnerVolumeSpecName "kube-api-access-mffmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:36.007478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.007449    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffmd\" (UniqueName: \"kubernetes.io/projected/702ee375-291c-4ed3-b5d1-981b14d795c4-kube-api-access-mffmd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.010539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.010472    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:46:36.108225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108187    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm\") pod \"2d0be272-d582-4a67-8956-c820a5ed5ed4\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") "
Oct 30 21:46:36.108446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108331    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache\") pod \"2d0be272-d582-4a67-8956-c820a5ed5ed4\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") "
Oct 30 21:46:36.108446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108395    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir\") pod \"2d0be272-d582-4a67-8956-c820a5ed5ed4\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") "
Oct 30 21:46:36.108571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108457    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs\") pod \"2d0be272-d582-4a67-8956-c820a5ed5ed4\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") "
Oct 30 21:46:36.108571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108537    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home\") pod \"2d0be272-d582-4a67-8956-c820a5ed5ed4\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") "
Oct 30 21:46:36.108696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108590    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw\" (UniqueName: \"kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw\") pod \"2d0be272-d582-4a67-8956-c820a5ed5ed4\" (UID: \"2d0be272-d582-4a67-8956-c820a5ed5ed4\") "
Oct 30 21:46:36.108696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108646    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache" pod "2d0be272-d582-4a67-8956-c820a5ed5ed4" (UID: "2d0be272-d582-4a67-8956-c820a5ed5ed4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:36.108696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108679    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir" pod "2d0be272-d582-4a67-8956-c820a5ed5ed4" (UID: "2d0be272-d582-4a67-8956-c820a5ed5ed4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:36.108918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.108896    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home" pod "2d0be272-d582-4a67-8956-c820a5ed5ed4" (UID: "2d0be272-d582-4a67-8956-c820a5ed5ed4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:36.109189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.109169    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.109277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.109194    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.109277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.109207    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.110954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.110928    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm" pod "2d0be272-d582-4a67-8956-c820a5ed5ed4" (UID: "2d0be272-d582-4a67-8956-c820a5ed5ed4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:36.110954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.110945    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw" pod "2d0be272-d582-4a67-8956-c820a5ed5ed4" (UID: "2d0be272-d582-4a67-8956-c820a5ed5ed4"). InnerVolumeSpecName "kube-api-access-8jcsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:36.111076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.110994    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs" pod "2d0be272-d582-4a67-8956-c820a5ed5ed4" (UID: "2d0be272-d582-4a67-8956-c820a5ed5ed4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:36.210702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.210663    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0be272-d582-4a67-8956-c820a5ed5ed4-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.210702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.210695    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jcsw\" (UniqueName: \"kubernetes.io/projected/2d0be272-d582-4a67-8956-c820a5ed5ed4-kube-api-access-8jcsw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.210917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.210710    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0be272-d582-4a67-8956-c820a5ed5ed4-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:36.533842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.533804    6695 generic.go:356] "Generic (PLEG): container finished" podID="702ee375-291c-4ed3-b5d1-981b14d795c4" containerID="ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76" exitCode=137
Oct 30 21:46:36.534344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.533851    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:36.534344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.533899    6695 scope.go:122] "RemoveContainer" containerID="ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76"
Oct 30 21:46:36.535437 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.535414    6695 generic.go:356] "Generic (PLEG): container finished" podID="2d0be272-d582-4a67-8956-c820a5ed5ed4" containerID="42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb" exitCode=0
Oct 30 21:46:36.535555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.535444    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" event={"ID":"2d0be272-d582-4a67-8956-c820a5ed5ed4","Type":"ContainerDied","Data":"42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb"}
Oct 30 21:46:36.535555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.535460    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"
Oct 30 21:46:36.535555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.535479    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2" event={"ID":"2d0be272-d582-4a67-8956-c820a5ed5ed4","Type":"ContainerDied","Data":"5de3b5e35a43421b4bded1eb5eeb6bef0e0c61d6f705da1cbaebd7ea52eeae6d"}
Oct 30 21:46:36.545955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.545749    6695 scope.go:122] "RemoveContainer" containerID="ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76"
Oct 30 21:46:36.546224 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:36.546050    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76\": container with ID starting with ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76 not found: ID does not exist" containerID="ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76"
Oct 30 21:46:36.546224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.546096    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76"} err="failed to get container status \"ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76\": rpc error: code = NotFound desc = could not find container \"ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76\": container with ID starting with ff821b03c1933d8b8b7f3246628efe0b03e84ba7220cfcd946e4179794070d76 not found: ID does not exist"
Oct 30 21:46:36.546224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.546119    6695 scope.go:122] "RemoveContainer" containerID="42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb"
Oct 30 21:46:36.555711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.555694    6695 scope.go:122] "RemoveContainer" containerID="42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb"
Oct 30 21:46:36.556001 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:36.555982    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb\": container with ID starting with 42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb not found: ID does not exist" containerID="42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb"
Oct 30 21:46:36.556040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.556009    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb"} err="failed to get container status \"42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb\": rpc error: code = NotFound desc = could not find container \"42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb\": container with ID starting with 42cf850da0ad008c3b33c09ac2c03ccb2593ce1ff5182887ba0293ad0b9370eb not found: ID does not exist"
Oct 30 21:46:36.586679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.586645    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"]
Oct 30 21:46:36.596007 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:36.595970    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-024fac\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:36.597496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.597467    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-bfabe1/auth-test-model-bfabe1-kserve-897c4dcc7-lxvt2"]
Oct 30 21:46:36.786847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:36.786766    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:36.799515 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:36.799476    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-024fac\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:37.223012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.222918    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0be272-d582-4a67-8956-c820a5ed5ed4" path="/var/lib/kubelet/pods/2d0be272-d582-4a67-8956-c820a5ed5ed4/volumes"
Oct 30 21:46:37.223429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.223406    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" path="/var/lib/kubelet/pods/702ee375-291c-4ed3-b5d1-981b14d795c4/volumes"
Oct 30 21:46:37.231046 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:37.231015    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7403-024fac\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="702ee375-291c-4ed3-b5d1-981b14d795c4" pod="opendatahub/test-curl-7403-024fac"
Oct 30 21:46:37.545441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.545380    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:37.546532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.546514    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:46:37.789653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.789621    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"]
Oct 30 21:46:37.790237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.790224    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2d0be272-d582-4a67-8956-c820a5ed5ed4" containerName="main"
Oct 30 21:46:37.790284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.790238    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0be272-d582-4a67-8956-c820a5ed5ed4" containerName="main"
Oct 30 21:46:37.790374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.790365    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0be272-d582-4a67-8956-c820a5ed5ed4" containerName="main"
Oct 30 21:46:37.795705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.795641    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:37.809939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.809913    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:37.810063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.809947    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:37.810063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.810002    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:46:37.812184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.812163    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:37.813366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.813307    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"]
Oct 30 21:46:37.937493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.937462    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:37.937717 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.937697    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:37.937846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:37.937824    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4bw\" (UniqueName: \"kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.002305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.002273    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"]
Oct 30 21:46:38.006641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.006618    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.029843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.029808    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"]
Oct 30 21:46:38.039743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.039706    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4bw\" (UniqueName: \"kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.039932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.039832    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.040038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.040018    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.040592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.040568    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.043200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.043174    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.064002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.063926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4bw\" (UniqueName: \"kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw\") pod \"maas-api-revoke-sub-306c3e8e-kpdv2\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.109409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.109377    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:38.141952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.141909    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4xc\" (UniqueName: \"kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.142197 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.142169    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.142369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.142348    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.243207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.243172    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.243392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.243329    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4xc\" (UniqueName: \"kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.243459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.243422    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.243961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.243931    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.245926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.245905    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.263022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.262993    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4xc\" (UniqueName: \"kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc\") pod \"maas-api-revoke-sub-c9dcde3b-47ztd\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.320895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.320803    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:38.422630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.422587    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"]
Oct 30 21:46:38.424855 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:38.424790    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14aa15de_e066_4c2c_af37_f2ae253271f7.slice/crio-7b0e701ceee296ab59657853364be98009ba608144e905419696f73b9c777ed3 WatchSource:0}: Error finding container 7b0e701ceee296ab59657853364be98009ba608144e905419696f73b9c777ed3: Status 404 returned error can't find the container with id 7b0e701ceee296ab59657853364be98009ba608144e905419696f73b9c777ed3
Oct 30 21:46:38.556598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.556560    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2" event={"ID":"14aa15de-e066-4c2c-af37-f2ae253271f7","Type":"ContainerStarted","Data":"7b0e701ceee296ab59657853364be98009ba608144e905419696f73b9c777ed3"}
Oct 30 21:46:38.628911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:38.628829    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"]
Oct 30 21:46:38.633018 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:38.632981    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1c8bb7_f75a_4a6f_bb17_602297849595.slice/crio-c9d66fcfc7bbb605e3336988a13c5b48d313b676409912ae69555b499d0da578 WatchSource:0}: Error finding container c9d66fcfc7bbb605e3336988a13c5b48d313b676409912ae69555b499d0da578: Status 404 returned error can't find the container with id c9d66fcfc7bbb605e3336988a13c5b48d313b676409912ae69555b499d0da578
Oct 30 21:46:39.582314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.571712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd" event={"ID":"5b1c8bb7-f75a-4a6f-bb17-602297849595","Type":"ContainerStarted","Data":"7aed659b9721f7eb4fe5c615b26cca74b413093a2513c128a356cad8d434fc1d"}
Oct 30 21:46:39.582314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.572549    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd" event={"ID":"5b1c8bb7-f75a-4a6f-bb17-602297849595","Type":"ContainerStarted","Data":"c9d66fcfc7bbb605e3336988a13c5b48d313b676409912ae69555b499d0da578"}
Oct 30 21:46:39.582314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.576102    6695 generic.go:356] "Generic (PLEG): container finished" podID="14aa15de-e066-4c2c-af37-f2ae253271f7" containerID="e311d0e41d82d378f79a93181521ce59060cb8deee4466247ae0cad14a51abc7" exitCode=0
Oct 30 21:46:39.582314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.577106    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2" event={"ID":"14aa15de-e066-4c2c-af37-f2ae253271f7","Type":"ContainerDied","Data":"e311d0e41d82d378f79a93181521ce59060cb8deee4466247ae0cad14a51abc7"}
Oct 30 21:46:39.619008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.618959    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd" podStartSLOduration=2.618946264 podStartE2EDuration="2.618946264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:39.618501704 +0000 UTC m=+2182.540960001" watchObservedRunningTime="2026-10-30 21:46:39.618946264 +0000 UTC m=+2182.541404562"
Oct 30 21:46:39.813929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.813888    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"]
Oct 30 21:46:39.832490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.832405    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:39.840219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.840181    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"]
Oct 30 21:46:39.846981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.846949    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c19461\""
Oct 30 21:46:39.846981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.846958    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c19461-dockercfg-ghr2z\""
Oct 30 21:46:39.903235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.902901    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"]
Oct 30 21:46:39.965627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.961008    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:39.967211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.966902    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pl9\" (UniqueName: \"kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9\") pod \"payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:39.967211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:39.967049    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.073807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.073761    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"]
Oct 30 21:46:40.084711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.080588    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.084711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.080661    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.084711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.080814    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csr7c\" (UniqueName: \"kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c\") pod \"payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.084711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.080919    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pl9\" (UniqueName: \"kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9\") pod \"payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.084711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.082226    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.101072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.101043    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pl9\" (UniqueName: \"kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9\") pod \"payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.149286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.149235    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.182556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.182523    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.182740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.182690    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csr7c\" (UniqueName: \"kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c\") pod \"payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.183546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.183521    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.206302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.206269    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csr7c\" (UniqueName: \"kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c\") pod \"payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.414883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.414806    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:40.457891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.457855    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"]
Oct 30 21:46:40.462055 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:40.462019    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf18df61_47dc_493e_957c_7c004dca33d8.slice/crio-a4cc90b17cd362132a5050549b6e99d5af5e67fea6be1bd82d1221e79bb29702 WatchSource:0}: Error finding container a4cc90b17cd362132a5050549b6e99d5af5e67fea6be1bd82d1221e79bb29702: Status 404 returned error can't find the container with id a4cc90b17cd362132a5050549b6e99d5af5e67fea6be1bd82d1221e79bb29702
Oct 30 21:46:40.585707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.585667    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" event={"ID":"bf18df61-47dc-493e-957c-7c004dca33d8","Type":"ContainerStarted","Data":"cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a"}
Oct 30 21:46:40.586188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.585738    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" event={"ID":"bf18df61-47dc-493e-957c-7c004dca33d8","Type":"ContainerStarted","Data":"a4cc90b17cd362132a5050549b6e99d5af5e67fea6be1bd82d1221e79bb29702"}
Oct 30 21:46:40.586188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.585918    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:40.587697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.587670    6695 generic.go:356] "Generic (PLEG): container finished" podID="5b1c8bb7-f75a-4a6f-bb17-602297849595" containerID="7aed659b9721f7eb4fe5c615b26cca74b413093a2513c128a356cad8d434fc1d" exitCode=0
Oct 30 21:46:40.587841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.587747    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd" event={"ID":"5b1c8bb7-f75a-4a6f-bb17-602297849595","Type":"ContainerDied","Data":"7aed659b9721f7eb4fe5c615b26cca74b413093a2513c128a356cad8d434fc1d"}
Oct 30 21:46:40.625875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.625800    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" podStartSLOduration=1.625776783 podStartE2EDuration="1.625776783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:40.624118848 +0000 UTC m=+2183.546577178" watchObservedRunningTime="2026-10-30 21:46:40.625776783 +0000 UTC m=+2183.548235082"
Oct 30 21:46:40.729224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.729190    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"]
Oct 30 21:46:40.730719 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:40.730687    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f65d1a_3a47_44ce_9a6b_37128be4a3f0.slice/crio-9b9c694429f9967becd773118af5967599a7f81adea9082683f539499fb7325a WatchSource:0}: Error finding container 9b9c694429f9967becd773118af5967599a7f81adea9082683f539499fb7325a: Status 404 returned error can't find the container with id 9b9c694429f9967becd773118af5967599a7f81adea9082683f539499fb7325a
Oct 30 21:46:40.793419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.793386    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:40.892269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.892236    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca\") pod \"14aa15de-e066-4c2c-af37-f2ae253271f7\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") "
Oct 30 21:46:40.892767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.892499    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca\") pod \"14aa15de-e066-4c2c-af37-f2ae253271f7\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") "
Oct 30 21:46:40.892767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.892597    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw\" (UniqueName: \"kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw\") pod \"14aa15de-e066-4c2c-af37-f2ae253271f7\" (UID: \"14aa15de-e066-4c2c-af37-f2ae253271f7\") "
Oct 30 21:46:40.893307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.893120    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca" pod "14aa15de-e066-4c2c-af37-f2ae253271f7" (UID: "14aa15de-e066-4c2c-af37-f2ae253271f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:40.893307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.893282    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:40.894975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.894946    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca" pod "14aa15de-e066-4c2c-af37-f2ae253271f7" (UID: "14aa15de-e066-4c2c-af37-f2ae253271f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:40.895314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.895290    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw" pod "14aa15de-e066-4c2c-af37-f2ae253271f7" (UID: "14aa15de-e066-4c2c-af37-f2ae253271f7"). InnerVolumeSpecName "kube-api-access-vs4bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:40.994495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.994399    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4bw\" (UniqueName: \"kubernetes.io/projected/14aa15de-e066-4c2c-af37-f2ae253271f7-kube-api-access-vs4bw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:40.994495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:40.994435    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14aa15de-e066-4c2c-af37-f2ae253271f7-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:41.625981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.625938    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" event={"ID":"20f65d1a-3a47-44ce-9a6b-37128be4a3f0","Type":"ContainerStarted","Data":"f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed"}
Oct 30 21:46:41.626490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.626010    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" event={"ID":"20f65d1a-3a47-44ce-9a6b-37128be4a3f0","Type":"ContainerStarted","Data":"9b9c694429f9967becd773118af5967599a7f81adea9082683f539499fb7325a"}
Oct 30 21:46:41.626490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.626043    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:41.627686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.627663    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"
Oct 30 21:46:41.627819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.627702    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2" event={"ID":"14aa15de-e066-4c2c-af37-f2ae253271f7","Type":"ContainerDied","Data":"7b0e701ceee296ab59657853364be98009ba608144e905419696f73b9c777ed3"}
Oct 30 21:46:41.627819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.627752    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0e701ceee296ab59657853364be98009ba608144e905419696f73b9c777ed3"
Oct 30 21:46:41.668565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.666734    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" podStartSLOduration=2.666713828 podStartE2EDuration="2.666713828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:41.662798033 +0000 UTC m=+2184.585256322" watchObservedRunningTime="2026-10-30 21:46:41.666713828 +0000 UTC m=+2184.589172128"
Oct 30 21:46:41.860384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:41.860062    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:42.028863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.028826    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca\") pod \"5b1c8bb7-f75a-4a6f-bb17-602297849595\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") "
Oct 30 21:46:42.029050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.028992    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc\" (UniqueName: \"kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc\") pod \"5b1c8bb7-f75a-4a6f-bb17-602297849595\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") "
Oct 30 21:46:42.029050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.029022    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca\") pod \"5b1c8bb7-f75a-4a6f-bb17-602297849595\" (UID: \"5b1c8bb7-f75a-4a6f-bb17-602297849595\") "
Oct 30 21:46:42.029204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.029181    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca" pod "5b1c8bb7-f75a-4a6f-bb17-602297849595" (UID: "5b1c8bb7-f75a-4a6f-bb17-602297849595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:42.029489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.029473    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:42.031166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.031139    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca" pod "5b1c8bb7-f75a-4a6f-bb17-602297849595" (UID: "5b1c8bb7-f75a-4a6f-bb17-602297849595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:42.031318 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.031258    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc" pod "5b1c8bb7-f75a-4a6f-bb17-602297849595" (UID: "5b1c8bb7-f75a-4a6f-bb17-602297849595"). InnerVolumeSpecName "kube-api-access-fw4xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:42.130482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.130449    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4xc\" (UniqueName: \"kubernetes.io/projected/5b1c8bb7-f75a-4a6f-bb17-602297849595-kube-api-access-fw4xc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:42.130482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.130480    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1c8bb7-f75a-4a6f-bb17-602297849595-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:42.641343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.641315    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"
Oct 30 21:46:42.641844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.641353    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd" event={"ID":"5b1c8bb7-f75a-4a6f-bb17-602297849595","Type":"ContainerDied","Data":"c9d66fcfc7bbb605e3336988a13c5b48d313b676409912ae69555b499d0da578"}
Oct 30 21:46:42.641844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:42.641411    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d66fcfc7bbb605e3336988a13c5b48d313b676409912ae69555b499d0da578"
Oct 30 21:46:45.525707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.525669    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7412-f545b7"]
Oct 30 21:46:45.526351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.526337    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5b1c8bb7-f75a-4a6f-bb17-602297849595" containerName="revoke-keys"
Oct 30 21:46:45.526410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.526353    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1c8bb7-f75a-4a6f-bb17-602297849595" containerName="revoke-keys"
Oct 30 21:46:45.526410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.526367    6695 cpu_manager.go:414] "containerMap: removing container" podUID="14aa15de-e066-4c2c-af37-f2ae253271f7" containerName="revoke-keys"
Oct 30 21:46:45.526410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.526372    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14aa15de-e066-4c2c-af37-f2ae253271f7" containerName="revoke-keys"
Oct 30 21:46:45.526505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.526489    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1c8bb7-f75a-4a6f-bb17-602297849595" containerName="revoke-keys"
Oct 30 21:46:45.526505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.526500    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="14aa15de-e066-4c2c-af37-f2ae253271f7" containerName="revoke-keys"
Oct 30 21:46:45.529961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.529940    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:45.543940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.543916    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7412-f545b7"]
Oct 30 21:46:45.551682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.551653    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vfm5w\""
Oct 30 21:46:45.686319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.686282    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwwg\" (UniqueName: \"kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg\") pod \"test-curl-7412-f545b7\" (UID: \"60edb819-060d-4a59-8099-c8eebb0d5dbb\") " pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:45.787831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.787737    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwwg\" (UniqueName: \"kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg\") pod \"test-curl-7412-f545b7\" (UID: \"60edb819-060d-4a59-8099-c8eebb0d5dbb\") " pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:45.807948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.807913    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwwg\" (UniqueName: \"kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg\") pod \"test-curl-7412-f545b7\" (UID: \"60edb819-060d-4a59-8099-c8eebb0d5dbb\") " pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:45.842519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:45.842481    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:46.192814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:46.192738    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7412-f545b7"]
Oct 30 21:46:46.195743 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:46.195714    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60edb819_060d_4a59_8099_c8eebb0d5dbb.slice/crio-75a7b39a24cedb14b740ed31f1d631b177f2d9f47037b7c6e05f25d3b5bd9343 WatchSource:0}: Error finding container 75a7b39a24cedb14b740ed31f1d631b177f2d9f47037b7c6e05f25d3b5bd9343: Status 404 returned error can't find the container with id 75a7b39a24cedb14b740ed31f1d631b177f2d9f47037b7c6e05f25d3b5bd9343
Oct 30 21:46:46.676582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:46.676550    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7412-f545b7" event={"ID":"60edb819-060d-4a59-8099-c8eebb0d5dbb","Type":"ContainerStarted","Data":"8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551"}
Oct 30 21:46:46.676988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:46.676591    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7412-f545b7" event={"ID":"60edb819-060d-4a59-8099-c8eebb0d5dbb","Type":"ContainerStarted","Data":"75a7b39a24cedb14b740ed31f1d631b177f2d9f47037b7c6e05f25d3b5bd9343"}
Oct 30 21:46:46.715245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:46.715190    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7412-f545b7" podStartSLOduration=1.715176269 podStartE2EDuration="1.715176269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:46.712386924 +0000 UTC m=+2189.634845221" watchObservedRunningTime="2026-10-30 21:46:46.715176269 +0000 UTC m=+2189.637634567"
Oct 30 21:46:47.543727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.543684    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7412-f545b7"]
Oct 30 21:46:47.547554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.547511    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7412-f545b7"]
Oct 30 21:46:47.762755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.762710    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"]
Oct 30 21:46:47.763492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.763471    6695 cpu_manager.go:414] "containerMap: removing container" podUID="60edb819-060d-4a59-8099-c8eebb0d5dbb" containerName="test-curl-7412-f545b7"
Oct 30 21:46:47.763540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.763494    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60edb819-060d-4a59-8099-c8eebb0d5dbb" containerName="test-curl-7412-f545b7"
Oct 30 21:46:47.763659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.763648    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="60edb819-060d-4a59-8099-c8eebb0d5dbb" containerName="test-curl-7412-f545b7"
Oct 30 21:46:47.767459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.767439    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.782640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.782612    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:47.782640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.782615    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:47.793712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.793686    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:47.793903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.793832    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:46:47.798543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.798516    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"]
Oct 30 21:46:47.816457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.816424    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.816457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.816462    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q55cj\" (UniqueName: \"kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.816704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.816550    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.918527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.918488    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.918527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.918530    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q55cj\" (UniqueName: \"kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.918815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.918579    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.919313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.919289    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.921147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.921125    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:47.937265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:47.937232    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q55cj\" (UniqueName: \"kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj\") pod \"maas-api-revoke-sub-e8a7fc2e-cvxdg\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:48.081420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:48.081336    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:48.388686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:48.388594    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"]
Oct 30 21:46:48.392043 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:48.392014    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e045e5_0553_4646_bb6f_de1fe94f8d31.slice/crio-df7363b27fd4d9647e03b58c8944c0eae30c23b33db605a584526998dcc0bfa3 WatchSource:0}: Error finding container df7363b27fd4d9647e03b58c8944c0eae30c23b33db605a584526998dcc0bfa3: Status 404 returned error can't find the container with id df7363b27fd4d9647e03b58c8944c0eae30c23b33db605a584526998dcc0bfa3
Oct 30 21:46:48.693199 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:48.693107    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg" event={"ID":"61e045e5-0553-4646-bb6f-de1fe94f8d31","Type":"ContainerStarted","Data":"df7363b27fd4d9647e03b58c8944c0eae30c23b33db605a584526998dcc0bfa3"}
Oct 30 21:46:48.693367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:48.693182    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7412-f545b7" podUID="60edb819-060d-4a59-8099-c8eebb0d5dbb" containerName="test-curl-7412-f545b7" containerID="cri-o://8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551" gracePeriod=2
Oct 30 21:46:48.706213 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:48.706178    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7412-f545b7\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="60edb819-060d-4a59-8099-c8eebb0d5dbb" pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:49.702687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:49.702567    6695 generic.go:356] "Generic (PLEG): container finished" podID="61e045e5-0553-4646-bb6f-de1fe94f8d31" containerID="448bf8814b33d29959b89e224f83c12ec402e76ceb4746b32836031e9bd84d87" exitCode=0
Oct 30 21:46:49.702687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:49.702644    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg" event={"ID":"61e045e5-0553-4646-bb6f-de1fe94f8d31","Type":"ContainerDied","Data":"448bf8814b33d29959b89e224f83c12ec402e76ceb4746b32836031e9bd84d87"}
Oct 30 21:46:51.014390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.014359    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:51.018243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.018220    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:51.177131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.177097    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg\" (UniqueName: \"kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg\") pod \"60edb819-060d-4a59-8099-c8eebb0d5dbb\" (UID: \"60edb819-060d-4a59-8099-c8eebb0d5dbb\") "
Oct 30 21:46:51.177336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.177197    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca\") pod \"61e045e5-0553-4646-bb6f-de1fe94f8d31\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") "
Oct 30 21:46:51.177336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.177237    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca\") pod \"61e045e5-0553-4646-bb6f-de1fe94f8d31\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") "
Oct 30 21:46:51.177336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.177301    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj\" (UniqueName: \"kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj\") pod \"61e045e5-0553-4646-bb6f-de1fe94f8d31\" (UID: \"61e045e5-0553-4646-bb6f-de1fe94f8d31\") "
Oct 30 21:46:51.177617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.177570    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca" pod "61e045e5-0553-4646-bb6f-de1fe94f8d31" (UID: "61e045e5-0553-4646-bb6f-de1fe94f8d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:51.177867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.177848    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:51.179301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.179277    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca" pod "61e045e5-0553-4646-bb6f-de1fe94f8d31" (UID: "61e045e5-0553-4646-bb6f-de1fe94f8d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:51.179397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.179372    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg" pod "60edb819-060d-4a59-8099-c8eebb0d5dbb" (UID: "60edb819-060d-4a59-8099-c8eebb0d5dbb"). InnerVolumeSpecName "kube-api-access-kjwwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:51.179434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.179390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj" pod "61e045e5-0553-4646-bb6f-de1fe94f8d31" (UID: "61e045e5-0553-4646-bb6f-de1fe94f8d31"). InnerVolumeSpecName "kube-api-access-q55cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:51.215979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.215943    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60edb819-060d-4a59-8099-c8eebb0d5dbb" path="/var/lib/kubelet/pods/60edb819-060d-4a59-8099-c8eebb0d5dbb/volumes"
Oct 30 21:46:51.278914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.278875    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjwwg\" (UniqueName: \"kubernetes.io/projected/60edb819-060d-4a59-8099-c8eebb0d5dbb-kube-api-access-kjwwg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:51.278914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.278906    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e045e5-0553-4646-bb6f-de1fe94f8d31-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:51.278914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.278917    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q55cj\" (UniqueName: \"kubernetes.io/projected/61e045e5-0553-4646-bb6f-de1fe94f8d31-kube-api-access-q55cj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:51.631934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.631901    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 21:46:51.665297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.665264    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7409-d28b5b"]
Oct 30 21:46:51.666022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.666007    6695 cpu_manager.go:414] "containerMap: removing container" podUID="61e045e5-0553-4646-bb6f-de1fe94f8d31" containerName="revoke-keys"
Oct 30 21:46:51.666087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.666023    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e045e5-0553-4646-bb6f-de1fe94f8d31" containerName="revoke-keys"
Oct 30 21:46:51.666170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.666161    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e045e5-0553-4646-bb6f-de1fe94f8d31" containerName="revoke-keys"
Oct 30 21:46:51.670034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.670018    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:51.683484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.683390    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftjg\" (UniqueName: \"kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg\") pod \"test-curl-7409-d28b5b\" (UID: \"d13f3e5b-4046-4153-9776-e8f5cc48b8bd\") " pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:51.685854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.685783    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7409-d28b5b"]
Oct 30 21:46:51.738941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.738906    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"
Oct 30 21:46:51.739091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.738939    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg" event={"ID":"61e045e5-0553-4646-bb6f-de1fe94f8d31","Type":"ContainerDied","Data":"df7363b27fd4d9647e03b58c8944c0eae30c23b33db605a584526998dcc0bfa3"}
Oct 30 21:46:51.739091 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.738987    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df7363b27fd4d9647e03b58c8944c0eae30c23b33db605a584526998dcc0bfa3"
Oct 30 21:46:51.746212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.746179    6695 generic.go:356] "Generic (PLEG): container finished" podID="60edb819-060d-4a59-8099-c8eebb0d5dbb" containerID="8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551" exitCode=137
Oct 30 21:46:51.746343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.746244    6695 scope.go:122] "RemoveContainer" containerID="8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551"
Oct 30 21:46:51.746343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.746271    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7412-f545b7"
Oct 30 21:46:51.756811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.756789    6695 scope.go:122] "RemoveContainer" containerID="8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551"
Oct 30 21:46:51.757128 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:51.757107    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551\": container with ID starting with 8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551 not found: ID does not exist" containerID="8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551"
Oct 30 21:46:51.757221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.757141    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551"} err="failed to get container status \"8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551\": rpc error: code = NotFound desc = could not find container \"8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551\": container with ID starting with 8438656c23aadff5d872abb697dedeaf6fdcf4fe8fd0e43df8b5e73dd2633551 not found: ID does not exist"
Oct 30 21:46:51.788643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.787892    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftjg\" (UniqueName: \"kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg\") pod \"test-curl-7409-d28b5b\" (UID: \"d13f3e5b-4046-4153-9776-e8f5cc48b8bd\") " pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:51.807731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.807702    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftjg\" (UniqueName: \"kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg\") pod \"test-curl-7409-d28b5b\" (UID: \"d13f3e5b-4046-4153-9776-e8f5cc48b8bd\") " pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:51.984952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:51.984863    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:52.321629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:52.321570    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7409-d28b5b"]
Oct 30 21:46:52.325521 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:52.325488    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd13f3e5b_4046_4153_9776_e8f5cc48b8bd.slice/crio-25ce7d7c57c32aefd647a641a37b190a5fbe2aefc3529b0660332e809fa40745 WatchSource:0}: Error finding container 25ce7d7c57c32aefd647a641a37b190a5fbe2aefc3529b0660332e809fa40745: Status 404 returned error can't find the container with id 25ce7d7c57c32aefd647a641a37b190a5fbe2aefc3529b0660332e809fa40745
Oct 30 21:46:52.645374 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:52.645294    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 21:46:52.755551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:52.755514    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7409-d28b5b" event={"ID":"d13f3e5b-4046-4153-9776-e8f5cc48b8bd","Type":"ContainerStarted","Data":"2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27"}
Oct 30 21:46:52.755774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:52.755563    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7409-d28b5b" event={"ID":"d13f3e5b-4046-4153-9776-e8f5cc48b8bd","Type":"ContainerStarted","Data":"25ce7d7c57c32aefd647a641a37b190a5fbe2aefc3529b0660332e809fa40745"}
Oct 30 21:46:52.799793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:52.799737    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7409-d28b5b" podStartSLOduration=1.799721348 podStartE2EDuration="1.799721348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:52.794635033 +0000 UTC m=+2195.717093323" watchObservedRunningTime="2026-10-30 21:46:52.799721348 +0000 UTC m=+2195.722179645"
Oct 30 21:46:53.667921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:53.667870    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7409-d28b5b"]
Oct 30 21:46:53.671442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:53.671413    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7409-d28b5b"]
Oct 30 21:46:54.772037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:54.771998    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7409-d28b5b" podUID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" containerName="test-curl-7409-d28b5b" containerID="cri-o://2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27" gracePeriod=2
Oct 30 21:46:54.785129 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:54.785096    6695 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7409-d28b5b\" is forbidden: User \"system:node:ip-10-0-142-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-232.ec2.internal' and this object" podUID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:57.015311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.015282    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:57.102876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.102787    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg\" (UniqueName: \"kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg\") pod \"d13f3e5b-4046-4153-9776-e8f5cc48b8bd\" (UID: \"d13f3e5b-4046-4153-9776-e8f5cc48b8bd\") "
Oct 30 21:46:57.104951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.104917    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg" pod "d13f3e5b-4046-4153-9776-e8f5cc48b8bd" (UID: "d13f3e5b-4046-4153-9776-e8f5cc48b8bd"). InnerVolumeSpecName "kube-api-access-bftjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:57.204876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.204833    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftjg\" (UniqueName: \"kubernetes.io/projected/d13f3e5b-4046-4153-9776-e8f5cc48b8bd-kube-api-access-bftjg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:46:57.216573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.216537    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" path="/var/lib/kubelet/pods/d13f3e5b-4046-4153-9776-e8f5cc48b8bd/volumes"
Oct 30 21:46:57.797748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.797708    6695 generic.go:356] "Generic (PLEG): container finished" podID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" containerID="2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27" exitCode=137
Oct 30 21:46:57.797961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.797768    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7409-d28b5b"
Oct 30 21:46:57.797961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.797844    6695 scope.go:122] "RemoveContainer" containerID="2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27"
Oct 30 21:46:57.809670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.809645    6695 scope.go:122] "RemoveContainer" containerID="2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27"
Oct 30 21:46:57.809964 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:46:57.809946    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27\": container with ID starting with 2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27 not found: ID does not exist" containerID="2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27"
Oct 30 21:46:57.810034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:57.809972    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27"} err="failed to get container status \"2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27\": rpc error: code = NotFound desc = could not find container \"2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27\": container with ID starting with 2bf957a670de89b341142df37f2e2227f3d9864f965d077677a6522da467db27 not found: ID does not exist"
Oct 30 21:46:58.165499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.165421    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"]
Oct 30 21:46:58.166133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.166115    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" containerName="test-curl-7409-d28b5b"
Oct 30 21:46:58.166186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.166134    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" containerName="test-curl-7409-d28b5b"
Oct 30 21:46:58.166268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.166258    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13f3e5b-4046-4153-9776-e8f5cc48b8bd" containerName="test-curl-7409-d28b5b"
Oct 30 21:46:58.171499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.171474    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.193356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.193321    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"]
Oct 30 21:46:58.197492 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.197463    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:58.197712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.197565    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:46:58.197712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.197583    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:58.197924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.197569    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:58.224235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.224199    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.224434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.224395    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5mr\" (UniqueName: \"kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.224507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.224456    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.325633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.325584    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.325815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.325712    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5mr\" (UniqueName: \"kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.325815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.325763    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.326395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.326369    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.328032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.328002    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.344564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.344538    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5mr\" (UniqueName: \"kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr\") pod \"maas-api-revoke-sub-a21b8900-42mm4\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.487232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.487147    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:46:58.801810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.801768    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"]
Oct 30 21:46:58.811149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:58.811111    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"]
Oct 30 21:46:58.811486 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:46:58.811101    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dff4467_a1c4_4d82_8554_f0fa23075681.slice/crio-7ef746a577a9f35770ab31a33ea005dcee3f2ae0090f0e3e0e2ea0c9b153933e WatchSource:0}: Error finding container 7ef746a577a9f35770ab31a33ea005dcee3f2ae0090f0e3e0e2ea0c9b153933e: Status 404 returned error can't find the container with id 7ef746a577a9f35770ab31a33ea005dcee3f2ae0090f0e3e0e2ea0c9b153933e
Oct 30 21:46:59.824450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:59.824408    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4" event={"ID":"3dff4467-a1c4-4d82-8554-f0fa23075681","Type":"ContainerStarted","Data":"a27073e776c294281dd4b37c7af202ad43877923a45b119ae033ef0ec356ab36"}
Oct 30 21:46:59.824900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:59.824540    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4" event={"ID":"3dff4467-a1c4-4d82-8554-f0fa23075681","Type":"ContainerStarted","Data":"7ef746a577a9f35770ab31a33ea005dcee3f2ae0090f0e3e0e2ea0c9b153933e"}
Oct 30 21:46:59.865412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:46:59.865355    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4" podStartSLOduration=1.86533957 podStartE2EDuration="1.86533957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:59.862205005 +0000 UTC m=+2202.784663303" watchObservedRunningTime="2026-10-30 21:46:59.86533957 +0000 UTC m=+2202.787797872"
Oct 30 21:47:00.119508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.119418    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"]
Oct 30 21:47:00.124333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.124304    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.156488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.154897    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3bc8ea-openshift-default-dockercfg-mxgms\""
Oct 30 21:47:00.156488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.155891    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"]
Oct 30 21:47:00.188596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.188544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.188829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.188796    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.188922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.188865    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcb6\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.189014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.188983    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.189093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.189046    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.189093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.189081    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.189257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.189235    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.189326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.189303    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.189385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.189331    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292081    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292158    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcb6\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292212    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292255    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292290    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292388    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292453    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292548    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.292904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.292581    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.293874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.293741    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.294048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.294021    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.294289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.294265    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.294684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.294659    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.296635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.296615    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.296830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.296706    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.312551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.312525    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcb6\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.316013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.315981    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token\") pod \"e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") " pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.444106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.444013    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:00.766458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.766427    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"]
Oct 30 21:47:00.769236 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:00.769205    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1552f927_4527_4263_bcea_e543aeb57e73.slice/crio-1b49f361ce6a75638fdf773725a904a42bcf4095af4b108d45422f3d78fc3d72 WatchSource:0}: Error finding container 1b49f361ce6a75638fdf773725a904a42bcf4095af4b108d45422f3d78fc3d72: Status 404 returned error can't find the container with id 1b49f361ce6a75638fdf773725a904a42bcf4095af4b108d45422f3d78fc3d72
Oct 30 21:47:00.772006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.771973    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:00.772116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.772040    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:00.772116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.772068    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:00.835242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.835207    6695 generic.go:356] "Generic (PLEG): container finished" podID="3dff4467-a1c4-4d82-8554-f0fa23075681" containerID="a27073e776c294281dd4b37c7af202ad43877923a45b119ae033ef0ec356ab36" exitCode=0
Oct 30 21:47:00.835650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.835258    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4" event={"ID":"3dff4467-a1c4-4d82-8554-f0fa23075681","Type":"ContainerDied","Data":"a27073e776c294281dd4b37c7af202ad43877923a45b119ae033ef0ec356ab36"}
Oct 30 21:47:00.838128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:00.838094    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" event={"ID":"1552f927-4527-4263-bcea-e543aeb57e73","Type":"ContainerStarted","Data":"1b49f361ce6a75638fdf773725a904a42bcf4095af4b108d45422f3d78fc3d72"}
Oct 30 21:47:01.076498 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:47:01.076455    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:47:01.091131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.091098    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"]
Oct 30 21:47:01.095598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.095576    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.115422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.115389    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3d5d13\""
Oct 30 21:47:01.115644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.115446    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3d5d13-dockercfg-hjkrm\""
Oct 30 21:47:01.118578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.118550    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"]
Oct 30 21:47:01.291626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.250166    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.291626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.250242    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm58h\" (UniqueName: \"kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h\") pod \"payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.355939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.355791    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.356136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.355998    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm58h\" (UniqueName: \"kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h\") pod \"payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.358435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.358403    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.361184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.361159    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"]
Oct 30 21:47:01.379207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.379149    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"]
Oct 30 21:47:01.379420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.379285    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.384666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.384566    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm58h\" (UniqueName: \"kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h\") pod \"payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.416629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.414899    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:01.469368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.458920    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.469368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.468077    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7glp\" (UniqueName: \"kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp\") pod \"payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.572397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.571148    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.572397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.571267    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7glp\" (UniqueName: \"kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp\") pod \"payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.572701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.572489    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.590284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.590239    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7glp\" (UniqueName: \"kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp\") pod \"payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.763915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.763883    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"]
Oct 30 21:47:01.766909 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:01.766880    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cf8de1_c1a9_4167_afe0_d50d84f0ca15.slice/crio-395e764a8c0569656842a8d1be36fd8413b83994cd510476a2b687a478d1bc8c WatchSource:0}: Error finding container 395e764a8c0569656842a8d1be36fd8413b83994cd510476a2b687a478d1bc8c: Status 404 returned error can't find the container with id 395e764a8c0569656842a8d1be36fd8413b83994cd510476a2b687a478d1bc8c
Oct 30 21:47:01.770205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.770142    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:01.853923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.853806    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" event={"ID":"26cf8de1-c1a9-4167-afe0-d50d84f0ca15","Type":"ContainerStarted","Data":"395e764a8c0569656842a8d1be36fd8413b83994cd510476a2b687a478d1bc8c"}
Oct 30 21:47:01.858753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.858722    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" event={"ID":"1552f927-4527-4263-bcea-e543aeb57e73","Type":"ContainerStarted","Data":"64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"}
Oct 30 21:47:01.899925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:01.899858    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podStartSLOduration=1.899837048 podStartE2EDuration="1.899837048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:01.89912182 +0000 UTC m=+2204.821580144" watchObservedRunningTime="2026-10-30 21:47:01.899837048 +0000 UTC m=+2204.822295351"
Oct 30 21:47:02.023579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.023546    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:47:02.077771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.077736    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca\") pod \"3dff4467-a1c4-4d82-8554-f0fa23075681\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") "
Oct 30 21:47:02.077946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.077827    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca\") pod \"3dff4467-a1c4-4d82-8554-f0fa23075681\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") "
Oct 30 21:47:02.077946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.077916    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr\" (UniqueName: \"kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr\") pod \"3dff4467-a1c4-4d82-8554-f0fa23075681\" (UID: \"3dff4467-a1c4-4d82-8554-f0fa23075681\") "
Oct 30 21:47:02.078084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.078062    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca" pod "3dff4467-a1c4-4d82-8554-f0fa23075681" (UID: "3dff4467-a1c4-4d82-8554-f0fa23075681"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:02.078585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.078559    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:02.080620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.080575    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr" pod "3dff4467-a1c4-4d82-8554-f0fa23075681" (UID: "3dff4467-a1c4-4d82-8554-f0fa23075681"). InnerVolumeSpecName "kube-api-access-vp5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:02.080754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.080687    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca" pod "3dff4467-a1c4-4d82-8554-f0fa23075681" (UID: "3dff4467-a1c4-4d82-8554-f0fa23075681"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:02.089106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.089068    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"]
Oct 30 21:47:02.100465 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:02.100427    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f934b6_ab7e_4697_8041_e3923bbc98bc.slice/crio-9bc30b66c84d2448d3c6fdb0a91a8d8fb7d0408ec396a25639e0ac11637571d1 WatchSource:0}: Error finding container 9bc30b66c84d2448d3c6fdb0a91a8d8fb7d0408ec396a25639e0ac11637571d1: Status 404 returned error can't find the container with id 9bc30b66c84d2448d3c6fdb0a91a8d8fb7d0408ec396a25639e0ac11637571d1
Oct 30 21:47:02.179934 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.179903    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dff4467-a1c4-4d82-8554-f0fa23075681-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:02.180044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.179938    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5mr\" (UniqueName: \"kubernetes.io/projected/3dff4467-a1c4-4d82-8554-f0fa23075681-kube-api-access-vp5mr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:02.451671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.448450    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:02.460272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.459973    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:02.867581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.867550    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"
Oct 30 21:47:02.868181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.867543    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4" event={"ID":"3dff4467-a1c4-4d82-8554-f0fa23075681","Type":"ContainerDied","Data":"7ef746a577a9f35770ab31a33ea005dcee3f2ae0090f0e3e0e2ea0c9b153933e"}
Oct 30 21:47:02.868181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.867732    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ef746a577a9f35770ab31a33ea005dcee3f2ae0090f0e3e0e2ea0c9b153933e"
Oct 30 21:47:02.869570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.869540    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" event={"ID":"26cf8de1-c1a9-4167-afe0-d50d84f0ca15","Type":"ContainerStarted","Data":"8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341"}
Oct 30 21:47:02.869749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.869615    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:02.871375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.871347    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" event={"ID":"02f934b6-ab7e-4697-8041-e3923bbc98bc","Type":"ContainerStarted","Data":"edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8"}
Oct 30 21:47:02.871504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.871400    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" event={"ID":"02f934b6-ab7e-4697-8041-e3923bbc98bc","Type":"ContainerStarted","Data":"9bc30b66c84d2448d3c6fdb0a91a8d8fb7d0408ec396a25639e0ac11637571d1"}
Oct 30 21:47:02.871680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.871657    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:02.871760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.871702    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:02.873012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.872995    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:47:02.910452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:02.910401    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" podStartSLOduration=1.910385798 podStartE2EDuration="1.910385798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:02.909916276 +0000 UTC m=+2205.832374588" watchObservedRunningTime="2026-10-30 21:47:02.910385798 +0000 UTC m=+2205.832844095"
Oct 30 21:47:03.008353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:03.008297    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" podStartSLOduration=2.008277427 podStartE2EDuration="2.008277427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:03.006656092 +0000 UTC m=+2205.929114402" watchObservedRunningTime="2026-10-30 21:47:03.008277427 +0000 UTC m=+2205.930735725"
Oct 30 21:47:04.327789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:04.327752    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"]
Oct 30 21:47:04.328280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:04.327962    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy" containerID="cri-o://a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350" gracePeriod=30
Oct 30 21:47:06.332377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.332346    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"]
Oct 30 21:47:06.333105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.333090    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3dff4467-a1c4-4d82-8554-f0fa23075681" containerName="revoke-keys"
Oct 30 21:47:06.333161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.333106    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dff4467-a1c4-4d82-8554-f0fa23075681" containerName="revoke-keys"
Oct 30 21:47:06.333241 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.333232    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dff4467-a1c4-4d82-8554-f0fa23075681" containerName="revoke-keys"
Oct 30 21:47:06.337084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.337068    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.363930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.363885    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3bc8ea\""
Oct 30 21:47:06.364504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.364474    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"]
Oct 30 21:47:06.372166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.372136    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3bc8ea-dockercfg-f9nqc\""
Oct 30 21:47:06.534641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.533544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvrd\" (UniqueName: \"kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd\") pod \"payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.534641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.533585    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.635090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.635006    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvrd\" (UniqueName: \"kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd\") pod \"payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.635090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.635055    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.636001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.635977    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.655558 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.655516    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvrd\" (UniqueName: \"kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd\") pod \"payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:06.951471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:06.951382    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:07.276298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:07.276269    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"]
Oct 30 21:47:07.279741 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:07.279712    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70ba5b3_fcb2_4334_91ff_f2b7b1ab11aa.slice/crio-473aac1a1e18ab8a4f040f0cbd73818dbb4bfc8813ed92a6255f710d7492f658 WatchSource:0}: Error finding container 473aac1a1e18ab8a4f040f0cbd73818dbb4bfc8813ed92a6255f710d7492f658: Status 404 returned error can't find the container with id 473aac1a1e18ab8a4f040f0cbd73818dbb4bfc8813ed92a6255f710d7492f658
Oct 30 21:47:07.910649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:07.910565    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv" event={"ID":"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa","Type":"ContainerStarted","Data":"41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665"}
Oct 30 21:47:07.911135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:07.910671    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv" event={"ID":"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa","Type":"ContainerStarted","Data":"473aac1a1e18ab8a4f040f0cbd73818dbb4bfc8813ed92a6255f710d7492f658"}
Oct 30 21:47:07.911135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:07.910715    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:07.952525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:07.952371    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv" podStartSLOduration=1.952351253 podStartE2EDuration="1.952351253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:07.949956736 +0000 UTC m=+2210.872415035" watchObservedRunningTime="2026-10-30 21:47:07.952351253 +0000 UTC m=+2210.874809554"
Oct 30 21:47:09.632621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.632575    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:47:09.706240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706151    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.706240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706219    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.706537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706264    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.706537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706318    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.706537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706354    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.706537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706495    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.707039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706546    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.707039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706572    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:09.707039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706582    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.707039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.706753    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token\") pod \"61919737-bb02-4132-8865-c1b33a99b398\" (UID: \"61919737-bb02-4132-8865-c1b33a99b398\") "
Oct 30 21:47:09.707265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707089    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:09.707481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707454    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:09.707614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707582    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:09.707741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707715    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:09.707826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707710    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61919737-bb02-4132-8865-c1b33a99b398-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.707826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707769    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.707826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.707802    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.709164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.709140    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:09.709704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.709673    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:09.709841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.709823    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "kube-api-access-dnsxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:09.709950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.709930    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo" pod "61919737-bb02-4132-8865-c1b33a99b398" (UID: "61919737-bb02-4132-8865-c1b33a99b398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:47:09.809792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.809756    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.809792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.809789    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsxg\" (UniqueName: \"kubernetes.io/projected/61919737-bb02-4132-8865-c1b33a99b398-kube-api-access-dnsxg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.810031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.809802    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.810031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.809814    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61919737-bb02-4132-8865-c1b33a99b398-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.810031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.809828    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.810031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.809841    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61919737-bb02-4132-8865-c1b33a99b398-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:09.942715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.942676    6695 generic.go:356] "Generic (PLEG): container finished" podID="61919737-bb02-4132-8865-c1b33a99b398" containerID="a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350" exitCode=0
Oct 30 21:47:09.942943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.942745    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" event={"ID":"61919737-bb02-4132-8865-c1b33a99b398","Type":"ContainerDied","Data":"a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350"}
Oct 30 21:47:09.942943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.942798    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp" event={"ID":"61919737-bb02-4132-8865-c1b33a99b398","Type":"ContainerDied","Data":"ca78c92fa6bb4246d3c14d529869584b22adbe38090b3b2f8c70be175851d075"}
Oct 30 21:47:09.942943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.942832    6695 scope.go:122] "RemoveContainer" containerID="a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350"
Oct 30 21:47:09.943145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:09.943120    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"
Oct 30 21:47:10.059048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:10.059016    6695 scope.go:122] "RemoveContainer" containerID="a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350"
Oct 30 21:47:10.059826 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:47:10.059749    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350\": container with ID starting with a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350 not found: ID does not exist" containerID="a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350"
Oct 30 21:47:10.059826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:10.059792    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350"} err="failed to get container status \"a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350\": rpc error: code = NotFound desc = could not find container \"a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350\": container with ID starting with a147914b851e077e5a34070f6bc602070ef46301821040d79837cc8206723350 not found: ID does not exist"
Oct 30 21:47:10.064534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:10.064499    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"]
Oct 30 21:47:10.077971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:10.077928    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6knsp"]
Oct 30 21:47:11.055168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.055133    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"]
Oct 30 21:47:11.066737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.066701    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b7397-9fxwf"]
Oct 30 21:47:11.216259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.216225    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507fe9b0-408f-4ee8-b1c0-c55252a0e087" path="/var/lib/kubelet/pods/507fe9b0-408f-4ee8-b1c0-c55252a0e087/volumes"
Oct 30 21:47:11.216974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.216955    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61919737-bb02-4132-8865-c1b33a99b398" path="/var/lib/kubelet/pods/61919737-bb02-4132-8865-c1b33a99b398/volumes"
Oct 30 21:47:11.453040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.452957    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"]
Oct 30 21:47:11.453684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.453667    6695 cpu_manager.go:414] "containerMap: removing container" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy"
Oct 30 21:47:11.453754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.453685    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy"
Oct 30 21:47:11.453881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.453870    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="61919737-bb02-4132-8865-c1b33a99b398" containerName="istio-proxy"
Oct 30 21:47:11.459342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.459316    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.482191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.482151    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"]
Oct 30 21:47:11.482486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.482464    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-427a6e-dockercfg-6gbxd\""
Oct 30 21:47:11.482594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.482553    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-427a6e\""
Oct 30 21:47:11.564358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.545747    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.564358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.545842    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grxw\" (UniqueName: \"kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw\") pod \"payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.597624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.591834    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"]
Oct 30 21:47:11.606622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.599833    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:11.656830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.654632    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:11.656830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.654730    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.656830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.654806    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hlx\" (UniqueName: \"kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx\") pod \"payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:11.656830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.654834    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grxw\" (UniqueName: \"kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw\") pod \"payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.656830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.656166    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.718744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.699686    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"]
Oct 30 21:47:11.734657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.730425    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grxw\" (UniqueName: \"kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw\") pod \"payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.767673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.758568    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hlx\" (UniqueName: \"kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx\") pod \"payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:11.767673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.758872    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:11.767673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.759837    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:11.775644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.775591    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:11.782209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:11.782179    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hlx\" (UniqueName: \"kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx\") pod \"payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:12.000224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.000137    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:12.091585 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.091479    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"]
Oct 30 21:47:12.095209 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:12.095185    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf160d273_95ba_4579_a0f5_b463d27fdaf2.slice/crio-157f0501509a8ec0e693c6e47894d36e7d41440b7dbeaf0b1d74360d48e35ce1 WatchSource:0}: Error finding container 157f0501509a8ec0e693c6e47894d36e7d41440b7dbeaf0b1d74360d48e35ce1: Status 404 returned error can't find the container with id 157f0501509a8ec0e693c6e47894d36e7d41440b7dbeaf0b1d74360d48e35ce1
Oct 30 21:47:12.306452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.306418    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"]
Oct 30 21:47:12.309783 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:12.309750    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ce44aa_6dc9_47a3_98d2_5e55f569416a.slice/crio-9aae2650b5e943710b3b13253567fb8eb3cc47df746ce7c5beefe5168f835fb1 WatchSource:0}: Error finding container 9aae2650b5e943710b3b13253567fb8eb3cc47df746ce7c5beefe5168f835fb1: Status 404 returned error can't find the container with id 9aae2650b5e943710b3b13253567fb8eb3cc47df746ce7c5beefe5168f835fb1
Oct 30 21:47:12.969646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.969597    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" event={"ID":"f160d273-95ba-4579-a0f5-b463d27fdaf2","Type":"ContainerStarted","Data":"3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c"}
Oct 30 21:47:12.969839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.969672    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" event={"ID":"f160d273-95ba-4579-a0f5-b463d27fdaf2","Type":"ContainerStarted","Data":"157f0501509a8ec0e693c6e47894d36e7d41440b7dbeaf0b1d74360d48e35ce1"}
Oct 30 21:47:12.969839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.969702    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:12.971476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.971448    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" event={"ID":"c2ce44aa-6dc9-47a3-98d2-5e55f569416a","Type":"ContainerStarted","Data":"75767dca364cec633c0591e2f9439912e0de6563a39f213217dd504671be4d65"}
Oct 30 21:47:12.971627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.971495    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" event={"ID":"c2ce44aa-6dc9-47a3-98d2-5e55f569416a","Type":"ContainerStarted","Data":"9aae2650b5e943710b3b13253567fb8eb3cc47df746ce7c5beefe5168f835fb1"}
Oct 30 21:47:12.971627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:12.971521    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:13.032143 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.032086    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" podStartSLOduration=2.032071841 podStartE2EDuration="2.032071841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.027581277 +0000 UTC m=+2215.950039587" watchObservedRunningTime="2026-10-30 21:47:13.032071841 +0000 UTC m=+2215.954530138"
Oct 30 21:47:13.065681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.065628    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" podStartSLOduration=2.065599618 podStartE2EDuration="2.065599618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.064524549 +0000 UTC m=+2215.986982847" watchObservedRunningTime="2026-10-30 21:47:13.065599618 +0000 UTC m=+2215.988057952"
Oct 30 21:47:13.837276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.837232    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" podUID="1fbef89e-d921-4885-8cd1-850bcaf697cf" containerName="main" containerID="cri-o://d5920e4d825304908c90d7bb2bcf115b38a2be5f95de2652160975d4c272af2c" gracePeriod=45
Oct 30 21:47:13.882510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.882480    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 21:47:13.882687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.882547    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 21:47:13.993121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.993078    6695 generic.go:356] "Generic (PLEG): container finished" podID="1fbef89e-d921-4885-8cd1-850bcaf697cf" containerID="d5920e4d825304908c90d7bb2bcf115b38a2be5f95de2652160975d4c272af2c" exitCode=0
Oct 30 21:47:13.993502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:13.993467    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" event={"ID":"1fbef89e-d921-4885-8cd1-850bcaf697cf","Type":"ContainerDied","Data":"d5920e4d825304908c90d7bb2bcf115b38a2be5f95de2652160975d4c272af2c"}
Oct 30 21:47:14.146561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.146535    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:47:14.196672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196599    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache\") pod \"1fbef89e-d921-4885-8cd1-850bcaf697cf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") "
Oct 30 21:47:14.196889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm\") pod \"1fbef89e-d921-4885-8cd1-850bcaf697cf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") "
Oct 30 21:47:14.196889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196862    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home\") pod \"1fbef89e-d921-4885-8cd1-850bcaf697cf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") "
Oct 30 21:47:14.197022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196891    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh\" (UniqueName: \"kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh\") pod \"1fbef89e-d921-4885-8cd1-850bcaf697cf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") "
Oct 30 21:47:14.197022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196929    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir\") pod \"1fbef89e-d921-4885-8cd1-850bcaf697cf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") "
Oct 30 21:47:14.197022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196947    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache" pod "1fbef89e-d921-4885-8cd1-850bcaf697cf" (UID: "1fbef89e-d921-4885-8cd1-850bcaf697cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:14.197022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.196957    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs\") pod \"1fbef89e-d921-4885-8cd1-850bcaf697cf\" (UID: \"1fbef89e-d921-4885-8cd1-850bcaf697cf\") "
Oct 30 21:47:14.197254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.197099    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home" pod "1fbef89e-d921-4885-8cd1-850bcaf697cf" (UID: "1fbef89e-d921-4885-8cd1-850bcaf697cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:14.197254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.197225    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir" pod "1fbef89e-d921-4885-8cd1-850bcaf697cf" (UID: "1fbef89e-d921-4885-8cd1-850bcaf697cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:14.197723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.197703    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:14.197723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.197722    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:14.197975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.197732    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:14.199161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.199137    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm" pod "1fbef89e-d921-4885-8cd1-850bcaf697cf" (UID: "1fbef89e-d921-4885-8cd1-850bcaf697cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:14.199265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.199194    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs" pod "1fbef89e-d921-4885-8cd1-850bcaf697cf" (UID: "1fbef89e-d921-4885-8cd1-850bcaf697cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:14.199265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.199238    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh" pod "1fbef89e-d921-4885-8cd1-850bcaf697cf" (UID: "1fbef89e-d921-4885-8cd1-850bcaf697cf"). InnerVolumeSpecName "kube-api-access-ctmfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:14.299335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.299296    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbef89e-d921-4885-8cd1-850bcaf697cf-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:14.299335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.299330    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmfh\" (UniqueName: \"kubernetes.io/projected/1fbef89e-d921-4885-8cd1-850bcaf697cf-kube-api-access-ctmfh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:14.299335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:14.299341    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbef89e-d921-4885-8cd1-850bcaf697cf-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:15.002254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.002225    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"
Oct 30 21:47:15.002734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.002260    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" event={"ID":"1fbef89e-d921-4885-8cd1-850bcaf697cf","Type":"ContainerDied","Data":"9fbf5451dc55b46e66ec2693f917fa240ec0017064faaf2a86afcaf0d9a290b5"}
Oct 30 21:47:15.002734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.002311    6695 scope.go:122] "RemoveContainer" containerID="d5920e4d825304908c90d7bb2bcf115b38a2be5f95de2652160975d4c272af2c"
Oct 30 21:47:15.070973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.070930    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"]
Oct 30 21:47:15.086172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.086138    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf"]
Oct 30 21:47:15.107238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.107197    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-64d142/auth-test-model-64d142-kserve-574d444489-fq9kf" podUID="1fbef89e-d921-4885-8cd1-850bcaf697cf" containerName="main" probeResult="failure" output="Get \"https://10.134.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:15.217557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:15.217522    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fbef89e-d921-4885-8cd1-850bcaf697cf" path="/var/lib/kubelet/pods/1fbef89e-d921-4885-8cd1-850bcaf697cf/volumes"
Oct 30 21:47:18.925741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:18.925706    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:47:20.563617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:20.563575    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"]
Oct 30 21:47:21.963668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:21.963632    6695 scope.go:122] "RemoveContainer" containerID="b3a79bd164f90ffce0bb96073c8c06172e63c333526a163023e7a563644dad55"
Oct 30 21:47:23.997089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:23.997061    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 21:47:23.997483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:23.997450    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 21:47:31.074487 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:47:31.074443    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:47:35.588721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:35.588678    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" podUID="1b260740-d39b-484c-b3b9-3c82910f90e4" containerName="main" containerID="cri-o://615797fb7170449c0c048455df82e54f5b84654f3f2fb3e47a8f918177102fc8" gracePeriod=45
Oct 30 21:47:35.681886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:35.681845    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" podUID="1b260740-d39b-484c-b3b9-3c82910f90e4" containerName="main" probeResult="failure" output="Get \"https://10.134.1.16:8000/ready\": dial tcp 10.134.1.16:8000: connect: connection refused"
Oct 30 21:47:36.159948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.159913    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"]
Oct 30 21:47:36.162308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.162276    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1fbef89e-d921-4885-8cd1-850bcaf697cf" containerName="main"
Oct 30 21:47:36.162308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.162304    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbef89e-d921-4885-8cd1-850bcaf697cf" containerName="main"
Oct 30 21:47:36.162653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.162632    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbef89e-d921-4885-8cd1-850bcaf697cf" containerName="main"
Oct 30 21:47:36.175635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.169147    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"]
Oct 30 21:47:36.175635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.169262    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.199748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.199201    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-846656-openshift-default-dockercfg-sl8ph\""
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252362    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252413    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpj6\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252434    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252455    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252481    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252529    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252561    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252585    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.252820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.252660    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.264363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.264334    6695 generic.go:356] "Generic (PLEG): container finished" podID="1b260740-d39b-484c-b3b9-3c82910f90e4" containerID="615797fb7170449c0c048455df82e54f5b84654f3f2fb3e47a8f918177102fc8" exitCode=0
Oct 30 21:47:36.264655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.264426    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" event={"ID":"1b260740-d39b-484c-b3b9-3c82910f90e4","Type":"ContainerDied","Data":"615797fb7170449c0c048455df82e54f5b84654f3f2fb3e47a8f918177102fc8"}
Oct 30 21:47:36.264815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.264793    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r" event={"ID":"1b260740-d39b-484c-b3b9-3c82910f90e4","Type":"ContainerDied","Data":"82d08319f8cb01893d680361be95ef382c9783acaabcceb43af95d1c1ab058a6"}
Oct 30 21:47:36.264899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.264818    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d08319f8cb01893d680361be95ef382c9783acaabcceb43af95d1c1ab058a6"
Oct 30 21:47:36.267099 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.267079    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:47:36.353873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.353778    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache\") pod \"1b260740-d39b-484c-b3b9-3c82910f90e4\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") "
Oct 30 21:47:36.353873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.353836    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm\") pod \"1b260740-d39b-484c-b3b9-3c82910f90e4\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") "
Oct 30 21:47:36.354133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.353893    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs\") pod \"1b260740-d39b-484c-b3b9-3c82910f90e4\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") "
Oct 30 21:47:36.354133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.353975    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k\" (UniqueName: \"kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k\") pod \"1b260740-d39b-484c-b3b9-3c82910f90e4\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") "
Oct 30 21:47:36.354133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.354081    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache" pod "1b260740-d39b-484c-b3b9-3c82910f90e4" (UID: "1b260740-d39b-484c-b3b9-3c82910f90e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:36.354133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.354118    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir\") pod \"1b260740-d39b-484c-b3b9-3c82910f90e4\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") "
Oct 30 21:47:36.354372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.354192    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home\") pod \"1b260740-d39b-484c-b3b9-3c82910f90e4\" (UID: \"1b260740-d39b-484c-b3b9-3c82910f90e4\") "
Oct 30 21:47:36.354547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.354522    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir" pod "1b260740-d39b-484c-b3b9-3c82910f90e4" (UID: "1b260740-d39b-484c-b3b9-3c82910f90e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:36.354547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.354540    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home" pod "1b260740-d39b-484c-b3b9-3c82910f90e4" (UID: "1b260740-d39b-484c-b3b9-3c82910f90e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:36.354769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.354747    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.355368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.355341    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.355718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.355696    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.355873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.355849    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpj6\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.355945 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.355904    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.356014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.355946    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.356014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356006    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.356134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356118    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.356347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356190    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.356347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356235    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.356347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356312    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:36.356347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356325    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:36.356648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356408    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:36.356714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356667    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.357033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.356998    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.357160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.357085    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs" pod "1b260740-d39b-484c-b3b9-3c82910f90e4" (UID: "1b260740-d39b-484c-b3b9-3c82910f90e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:36.357222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.357200    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.357284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.357204    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k" pod "1b260740-d39b-484c-b3b9-3c82910f90e4" (UID: "1b260740-d39b-484c-b3b9-3c82910f90e4"). InnerVolumeSpecName "kube-api-access-pg89k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:36.357350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.357285    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.357517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.357502    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm" pod "1b260740-d39b-484c-b3b9-3c82910f90e4" (UID: "1b260740-d39b-484c-b3b9-3c82910f90e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:36.358891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.358854    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.360757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.360723    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.394462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.394423    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.394733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.394707    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpj6\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:36.457461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.457421    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b260740-d39b-484c-b3b9-3c82910f90e4-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:36.457461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.457452    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b260740-d39b-484c-b3b9-3c82910f90e4-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:36.457461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.457465    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg89k\" (UniqueName: \"kubernetes.io/projected/1b260740-d39b-484c-b3b9-3c82910f90e4-kube-api-access-pg89k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:36.587626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:36.584878    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:37.046627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.044962    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"]
Oct 30 21:47:37.094504 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:37.094469    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c656625_0f5d_422f_a718_c447c368fcf6.slice/crio-4d952e255e9c0c4d206d88cd1abd9a45a9d019686cff98cd1c9cd9ba8bb4dc17 WatchSource:0}: Error finding container 4d952e255e9c0c4d206d88cd1abd9a45a9d019686cff98cd1c9cd9ba8bb4dc17: Status 404 returned error can't find the container with id 4d952e255e9c0c4d206d88cd1abd9a45a9d019686cff98cd1c9cd9ba8bb4dc17
Oct 30 21:47:37.124053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.123993    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:37.124445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.124406    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:37.124658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.124638    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:37.411324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.406354    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"
Oct 30 21:47:37.411324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.407403    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" event={"ID":"8c656625-0f5d-422f-a718-c447c368fcf6","Type":"ContainerStarted","Data":"4d952e255e9c0c4d206d88cd1abd9a45a9d019686cff98cd1c9cd9ba8bb4dc17"}
Oct 30 21:47:37.532691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.532653    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"]
Oct 30 21:47:37.642681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:37.638548    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-5494e8/auth-test-model-5494e8-kserve-567cc45f45-4p29r"]
Oct 30 21:47:38.438870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:38.438828    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" event={"ID":"8c656625-0f5d-422f-a718-c447c368fcf6","Type":"ContainerStarted","Data":"4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"}
Oct 30 21:47:38.496207 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:38.496155    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podStartSLOduration=2.496139264 podStartE2EDuration="2.496139264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:38.494513422 +0000 UTC m=+2241.416971720" watchObservedRunningTime="2026-10-30 21:47:38.496139264 +0000 UTC m=+2241.418597561"
Oct 30 21:47:38.586043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:38.586003    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:38.591871 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:38.591838    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:39.251230 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:39.251192    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b260740-d39b-484c-b3b9-3c82910f90e4" path="/var/lib/kubelet/pods/1b260740-d39b-484c-b3b9-3c82910f90e4/volumes"
Oct 30 21:47:39.446826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:39.446796    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:39.447854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:39.447830    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:47:40.365553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:40.365514    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"]
Oct 30 21:47:42.565619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.565570    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"]
Oct 30 21:47:42.567648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.566631    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1b260740-d39b-484c-b3b9-3c82910f90e4" containerName="main"
Oct 30 21:47:42.567648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.566653    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b260740-d39b-484c-b3b9-3c82910f90e4" containerName="main"
Oct 30 21:47:42.567648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.566772    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b260740-d39b-484c-b3b9-3c82910f90e4" containerName="main"
Oct 30 21:47:42.572340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.572303    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.591554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.591527    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-805edc-openshift-default-dockercfg-vvclz\""
Oct 30 21:47:42.607664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.607629    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"]
Oct 30 21:47:42.662776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.662747    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.662981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.662816    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.662981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.662886    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.663104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.663004    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.663149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.663100    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.663219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.663198    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvt5x\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.663317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.663302    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.663397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.663361    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.663448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.663420    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.820751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.770507    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.821104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.821078    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.821769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.821740    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.822525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.822501    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.823484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.823461    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.824414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.824394    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.824931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.824910    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.825442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.825424    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvt5x\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.825976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.825956    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.833633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.824799    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.833872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.823069    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.834023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.825295    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.834135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.824247    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.834243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.822288    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.917643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.905755    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.919819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.919748    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.922269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.922188    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:42.932636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:42.928028    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvt5x\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:43.195133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:43.195099    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:43.549922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:43.549767    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"]
Oct 30 21:47:43.553793 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:43.553760    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6d8f36_f381_4ebd_a5a7_912a93c49842.slice/crio-a6e17784250dca1a151c2582bbee50321708bb82c27361c7122a4355905a8a1c WatchSource:0}: Error finding container a6e17784250dca1a151c2582bbee50321708bb82c27361c7122a4355905a8a1c: Status 404 returned error can't find the container with id a6e17784250dca1a151c2582bbee50321708bb82c27361c7122a4355905a8a1c
Oct 30 21:47:43.556412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:43.556381    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:43.556484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:43.556447    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:43.556484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:43.556478    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:44.492000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:44.491961    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" event={"ID":"bb6d8f36-f381-4ebd-a5a7-912a93c49842","Type":"ContainerStarted","Data":"921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"}
Oct 30 21:47:44.492469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:44.492029    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" event={"ID":"bb6d8f36-f381-4ebd-a5a7-912a93c49842","Type":"ContainerStarted","Data":"a6e17784250dca1a151c2582bbee50321708bb82c27361c7122a4355905a8a1c"}
Oct 30 21:47:44.574793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:44.574716    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podStartSLOduration=2.5746934870000002 podStartE2EDuration="2.574693487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:44.559933634 +0000 UTC m=+2247.482391958" watchObservedRunningTime="2026-10-30 21:47:44.574693487 +0000 UTC m=+2247.497151786"
Oct 30 21:47:45.206627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:45.200043    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:45.252809 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:45.252775    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:45.500968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:45.500887    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:45.501886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:45.501865    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:47:47.521188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:47.521150    6695 generic.go:356] "Generic (PLEG): container finished" podID="d0b41768-b63b-44df-94a1-292629eb7d29" containerID="7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2" exitCode=0
Oct 30 21:47:47.523018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:47.522990    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" event={"ID":"d0b41768-b63b-44df-94a1-292629eb7d29","Type":"ContainerDied","Data":"7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"}
Oct 30 21:47:47.523731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:47.523712    6695 scope.go:122] "RemoveContainer" containerID="7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"
Oct 30 21:47:47.526598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:47.526569    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:47.526725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:47.526677    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:47.526725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:47.526717    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:47:48.532574 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:48.532538    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" event={"ID":"d0b41768-b63b-44df-94a1-292629eb7d29","Type":"ContainerStarted","Data":"2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb"}
Oct 30 21:47:48.780065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:48.780024    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:47:48.781579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:48.781533    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body=
Oct 30 21:47:48.781813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:48.781622    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused"
Oct 30 21:47:49.304655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.293455    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"]
Oct 30 21:47:49.308654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.308580    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.450000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.394320    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-805edc\""
Oct 30 21:47:49.450260 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.397313    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-805edc-dockercfg-5gtlq\""
Oct 30 21:47:49.450396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.398867    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.452136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.452062    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwsj\" (UniqueName: \"kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj\") pod \"payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.474964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.458298    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"]
Oct 30 21:47:49.553630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.553581    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwsj\" (UniqueName: \"kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj\") pod \"payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.554153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.553819    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.554795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.554746    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.573577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.573546    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwsj\" (UniqueName: \"kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj\") pod \"payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.759315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.759248    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:49.780562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.780518    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body=
Oct 30 21:47:49.780730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:49.780614    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused"
Oct 30 21:47:50.100958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:50.100864    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"]
Oct 30 21:47:50.104262 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:47:50.104192    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3406f3a1_57d0_418a_99b0_2ef323fe49d6.slice/crio-ec3bf4dfd09f74b880e6ae31842d78bdc52041d1c3157c4d216830003d628022 WatchSource:0}: Error finding container ec3bf4dfd09f74b880e6ae31842d78bdc52041d1c3157c4d216830003d628022: Status 404 returned error can't find the container with id ec3bf4dfd09f74b880e6ae31842d78bdc52041d1c3157c4d216830003d628022
Oct 30 21:47:50.561689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:50.561649    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" event={"ID":"3406f3a1-57d0-418a-99b0-2ef323fe49d6","Type":"ContainerStarted","Data":"52de8f4e196b59d44947f5f309264cbe7c286df7a68bafff74ab9310395e9c4c"}
Oct 30 21:47:50.562232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:50.561892    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:47:50.562232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:50.561942    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" event={"ID":"3406f3a1-57d0-418a-99b0-2ef323fe49d6","Type":"ContainerStarted","Data":"ec3bf4dfd09f74b880e6ae31842d78bdc52041d1c3157c4d216830003d628022"}
Oct 30 21:47:50.787625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:50.783252    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body=
Oct 30 21:47:50.787625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:50.783333    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused"
Oct 30 21:47:51.780632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:51.780574    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body=
Oct 30 21:47:51.781057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:51.780677    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused"
Oct 30 21:47:52.545947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:52.545908    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:47:52.782957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:52.781001    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body=
Oct 30 21:47:52.782957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:52.781063    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused"
Oct 30 21:47:53.780831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:53.780793    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body=
Oct 30 21:47:53.781064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:53.780861    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused"
Oct 30 21:47:54.785981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:54.785945    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:47:54.787173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:54.787153    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 21:47:54.863507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:54.863451    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" podStartSLOduration=5.863435516 podStartE2EDuration="5.863435516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:50.646991179 +0000 UTC m=+2253.569449477" watchObservedRunningTime="2026-10-30 21:47:54.863435516 +0000 UTC m=+2257.785893814"
Oct 30 21:47:55.394627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.392843    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" podUID="7c5677f9-2262-49ac-9694-b71b2398cc88" containerName="main" containerID="cri-o://9d27d3a90bb5ba31fdc36e3c18ad209ddb08ab76208eae6e132f9b59969e68eb" gracePeriod=45
Oct 30 21:47:55.455006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.454965    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" podUID="7c5677f9-2262-49ac-9694-b71b2398cc88" containerName="main" probeResult="failure" output="Get \"https://10.134.1.18:8000/ready\": dial tcp 10.134.1.18:8000: connect: connection refused"
Oct 30 21:47:55.622985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.622569    6695 generic.go:356] "Generic (PLEG): container finished" podID="7c5677f9-2262-49ac-9694-b71b2398cc88" containerID="9d27d3a90bb5ba31fdc36e3c18ad209ddb08ab76208eae6e132f9b59969e68eb" exitCode=0
Oct 30 21:47:55.622985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.622895    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" event={"ID":"7c5677f9-2262-49ac-9694-b71b2398cc88","Type":"ContainerDied","Data":"9d27d3a90bb5ba31fdc36e3c18ad209ddb08ab76208eae6e132f9b59969e68eb"}
Oct 30 21:47:55.677863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.677835    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:47:55.822626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.813164    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir\") pod \"7c5677f9-2262-49ac-9694-b71b2398cc88\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") "
Oct 30 21:47:55.822626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.813279    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs\") pod \"7c5677f9-2262-49ac-9694-b71b2398cc88\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") "
Oct 30 21:47:55.822626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.813420    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home\") pod \"7c5677f9-2262-49ac-9694-b71b2398cc88\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") "
Oct 30 21:47:55.822626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.813462    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache\") pod \"7c5677f9-2262-49ac-9694-b71b2398cc88\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") "
Oct 30 21:47:55.822626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.813491    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t\") pod \"7c5677f9-2262-49ac-9694-b71b2398cc88\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") "
Oct 30 21:47:55.822626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.813626    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm\") pod \"7c5677f9-2262-49ac-9694-b71b2398cc88\" (UID: \"7c5677f9-2262-49ac-9694-b71b2398cc88\") "
Oct 30 21:47:55.877645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.855741    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir" pod "7c5677f9-2262-49ac-9694-b71b2398cc88" (UID: "7c5677f9-2262-49ac-9694-b71b2398cc88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:55.877645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.856895    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home" pod "7c5677f9-2262-49ac-9694-b71b2398cc88" (UID: "7c5677f9-2262-49ac-9694-b71b2398cc88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:55.877645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.857917    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache" pod "7c5677f9-2262-49ac-9694-b71b2398cc88" (UID: "7c5677f9-2262-49ac-9694-b71b2398cc88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:55.889903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.889793    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs" pod "7c5677f9-2262-49ac-9694-b71b2398cc88" (UID: "7c5677f9-2262-49ac-9694-b71b2398cc88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:55.914879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.909763    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm" pod "7c5677f9-2262-49ac-9694-b71b2398cc88" (UID: "7c5677f9-2262-49ac-9694-b71b2398cc88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:55.929659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.919846    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t" pod "7c5677f9-2262-49ac-9694-b71b2398cc88" (UID: "7c5677f9-2262-49ac-9694-b71b2398cc88"). InnerVolumeSpecName "kube-api-access-n564t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:55.953627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.951993    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:55.953627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.952033    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:55.953627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.952052    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n564t\" (UniqueName: \"kubernetes.io/projected/7c5677f9-2262-49ac-9694-b71b2398cc88-kube-api-access-n564t\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:55.953627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.952063    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:55.953627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.952076    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5677f9-2262-49ac-9694-b71b2398cc88-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:55.953627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:55.952089    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5677f9-2262-49ac-9694-b71b2398cc88-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:47:56.632764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:56.632730    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"
Oct 30 21:47:56.632956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:56.632731    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2" event={"ID":"7c5677f9-2262-49ac-9694-b71b2398cc88","Type":"ContainerDied","Data":"f40f8ab44162e79c960ab7dce49716c6c89bb499dcbe5e8871af15eb9cf0dac5"}
Oct 30 21:47:56.632956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:56.632878    6695 scope.go:122] "RemoveContainer" containerID="9d27d3a90bb5ba31fdc36e3c18ad209ddb08ab76208eae6e132f9b59969e68eb"
Oct 30 21:47:56.699816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:56.699725    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"]
Oct 30 21:47:56.708502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:56.708465    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-ae6492/auth-test-model-ae6492-kserve-865d4bffc7-9lvw2"]
Oct 30 21:47:57.238180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:47:57.238137    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c5677f9-2262-49ac-9694-b71b2398cc88" path="/var/lib/kubelet/pods/7c5677f9-2262-49ac-9694-b71b2398cc88/volumes"
Oct 30 21:48:01.079836 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:48:01.079660    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:48:01.573693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:01.573659    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 21:48:03.913816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:03.913766    6695 patch_prober.go:29] interesting pod/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:48:03.914355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:03.913848    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:48:07.725559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:07.725526    6695 generic.go:356] "Generic (PLEG): container finished" podID="1552f927-4527-4263-bcea-e543aeb57e73" containerID="64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3" exitCode=0
Oct 30 21:48:07.726065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:07.725612    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" event={"ID":"1552f927-4527-4263-bcea-e543aeb57e73","Type":"ContainerDied","Data":"64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"}
Oct 30 21:48:07.726124 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:07.726071    6695 scope.go:122] "RemoveContainer" containerID="64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"
Oct 30 21:48:07.727882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:07.727857    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:07.727969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:07.727924    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:07.727969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:07.727955    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:08.744183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:08.744127    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" event={"ID":"1552f927-4527-4263-bcea-e543aeb57e73","Type":"ContainerStarted","Data":"ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751"}
Oct 30 21:48:09.444789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:09.444758    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:48:09.444995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:09.444970    6695 patch_prober.go:29] interesting pod/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Oct 30 21:48:09.445050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:09.445030    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Oct 30 21:48:10.445334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:10.445298    6695 patch_prober.go:29] interesting pod/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Oct 30 21:48:10.445861 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:10.445361    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Oct 30 21:48:11.445301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:11.445264    6695 patch_prober.go:29] interesting pod/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Oct 30 21:48:11.445507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:11.445332    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Oct 30 21:48:12.445375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:12.445335    6695 patch_prober.go:29] interesting pod/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Oct 30 21:48:12.445583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:12.445389    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Oct 30 21:48:13.445432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:13.445392    6695 patch_prober.go:29] interesting pod/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Oct 30 21:48:13.445663 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:13.445462    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Oct 30 21:48:14.449274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:14.449241    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:48:14.449963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:14.449941    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:48:14.450706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:14.450683    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:48:18.586308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.586272    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"]
Oct 30 21:48:18.587564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.587538    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7c5677f9-2262-49ac-9694-b71b2398cc88" containerName="main"
Oct 30 21:48:18.587564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.587563    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c5677f9-2262-49ac-9694-b71b2398cc88" containerName="main"
Oct 30 21:48:18.587856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.587839    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c5677f9-2262-49ac-9694-b71b2398cc88" containerName="main"
Oct 30 21:48:18.592405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.592384    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.615824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.615790    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"]
Oct 30 21:48:18.621431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.621409    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3bc8ea\"/\"kube-root-ca.crt\""
Oct 30 21:48:18.621559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.621443    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3bc8ea\"/\"openshift-service-ca.crt\""
Oct 30 21:48:18.623517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.623498    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3bc8ea\"/\"default-dockercfg-rljfk\""
Oct 30 21:48:18.623864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.623758    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3bc8ea\"/\"autores-model-3bc8ea-kserve-self-signed-certs\""
Oct 30 21:48:18.684416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.684376    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.684660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.684509    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfdm\" (UniqueName: \"kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.684660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.684559    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.684660 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.684586    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.684815 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.684706    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.684879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.684870    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.785931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.785892    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfdm\" (UniqueName: \"kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.785931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.785934    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.785951    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.786006    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.786188    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.786337    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.786396    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.786426    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.786568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.786523    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.788520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.788495    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.788782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.788760    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.808975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.808948    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfdm\" (UniqueName: \"kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm\") pod \"autores-model-3bc8ea-kserve-5bf447457c-xtnnt\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") " pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:18.927692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:18.925864    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:19.285331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:19.285300    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"]
Oct 30 21:48:19.287590 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:48:19.287561    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4c6cc2_7690_43bb_82e2_52e36f1d14c5.slice/crio-4f6470ea64423f78c5a54176142a04a4fdeb0529ac91c55e119d448a49ccfacf WatchSource:0}: Error finding container 4f6470ea64423f78c5a54176142a04a4fdeb0529ac91c55e119d448a49ccfacf: Status 404 returned error can't find the container with id 4f6470ea64423f78c5a54176142a04a4fdeb0529ac91c55e119d448a49ccfacf
Oct 30 21:48:19.977909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:19.975076    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" event={"ID":"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5","Type":"ContainerStarted","Data":"201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1"}
Oct 30 21:48:19.977909 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:19.975139    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" event={"ID":"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5","Type":"ContainerStarted","Data":"4f6470ea64423f78c5a54176142a04a4fdeb0529ac91c55e119d448a49ccfacf"}
Oct 30 21:48:20.023086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:20.023035    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" podStartSLOduration=2.023020895 podStartE2EDuration="2.023020895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:20.014888874 +0000 UTC m=+2282.937347172" watchObservedRunningTime="2026-10-30 21:48:20.023020895 +0000 UTC m=+2282.945479190"
Oct 30 21:48:20.926783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:20.926735    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:21.396013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.395978    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"]
Oct 30 21:48:21.400725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.400701    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.428144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.428115    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fdb168-dockercfg-4vrm8\""
Oct 30 21:48:21.428392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.428126    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fdb168\""
Oct 30 21:48:21.434077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.434042    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"]
Oct 30 21:48:21.442638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.442587    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.442913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.442888    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b844\" (UniqueName: \"kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844\") pod \"payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.457644 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.457594    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"]
Oct 30 21:48:21.463420 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.463396    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.567629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.547230    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b844\" (UniqueName: \"kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844\") pod \"payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.567629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.547397    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.567629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.548518    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.659688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.658831    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.659688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.658982    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twzjq\" (UniqueName: \"kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq\") pod \"payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.672440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.672082    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"]
Oct 30 21:48:21.685675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.685511    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b844\" (UniqueName: \"kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844\") pod \"payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.716733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.716697    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:21.760112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.760039    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.760291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.760225    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twzjq\" (UniqueName: \"kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq\") pod \"payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.760910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.760888    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.786534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.786500    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twzjq\" (UniqueName: \"kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq\") pod \"payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:21.988876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:21.988798    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:22.025993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:22.025684    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"]
Oct 30 21:48:22.028051 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:48:22.028009    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0002ef74_d862_4bb3_9615_7ec896d884c4.slice/crio-44d4c628fe2fd18c5ae8cebc1af13a1936443958fef3681a337384c42719b453 WatchSource:0}: Error finding container 44d4c628fe2fd18c5ae8cebc1af13a1936443958fef3681a337384c42719b453: Status 404 returned error can't find the container with id 44d4c628fe2fd18c5ae8cebc1af13a1936443958fef3681a337384c42719b453
Oct 30 21:48:22.067246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:22.067109    6695 scope.go:122] "RemoveContainer" containerID="2c2c56d6f20c0ed68893423c5da86212a161bfee177f8b9d0a6ba04fa12f6d95"
Oct 30 21:48:22.092418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:22.092396    6695 scope.go:122] "RemoveContainer" containerID="737e365af361874fb7b1d1f796a8d5cec1a116448a97ad773413f61de26dee8d"
Oct 30 21:48:22.336666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:22.336563    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"]
Oct 30 21:48:22.339435 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:48:22.339406    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1702520_0360_4cbe_81dd_038186e3cb40.slice/crio-ff6334bf2e20dae9f0d4686eefb1a309f669d4d3d46578f32eac21cc2a5596fd WatchSource:0}: Error finding container ff6334bf2e20dae9f0d4686eefb1a309f669d4d3d46578f32eac21cc2a5596fd: Status 404 returned error can't find the container with id ff6334bf2e20dae9f0d4686eefb1a309f669d4d3d46578f32eac21cc2a5596fd
Oct 30 21:48:23.081616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.081556    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" event={"ID":"c1702520-0360-4cbe-81dd-038186e3cb40","Type":"ContainerStarted","Data":"e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14"}
Oct 30 21:48:23.082735 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.082707    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:23.082887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.082746    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" event={"ID":"c1702520-0360-4cbe-81dd-038186e3cb40","Type":"ContainerStarted","Data":"ff6334bf2e20dae9f0d4686eefb1a309f669d4d3d46578f32eac21cc2a5596fd"}
Oct 30 21:48:23.084493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.084464    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" event={"ID":"0002ef74-d862-4bb3-9615-7ec896d884c4","Type":"ContainerStarted","Data":"36cbf9275f38ee9397962b1508da04926ddbd0940760e49cc329fa1c154a0128"}
Oct 30 21:48:23.084627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.084516    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" event={"ID":"0002ef74-d862-4bb3-9615-7ec896d884c4","Type":"ContainerStarted","Data":"44d4c628fe2fd18c5ae8cebc1af13a1936443958fef3681a337384c42719b453"}
Oct 30 21:48:23.085694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.085365    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:23.124715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.124667    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" podStartSLOduration=2.124651751 podStartE2EDuration="2.124651751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:23.120492335 +0000 UTC m=+2286.042950633" watchObservedRunningTime="2026-10-30 21:48:23.124651751 +0000 UTC m=+2286.047110075"
Oct 30 21:48:23.159622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:23.159548    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" podStartSLOduration=2.159534286 podStartE2EDuration="2.159534286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:23.158599204 +0000 UTC m=+2286.081057503" watchObservedRunningTime="2026-10-30 21:48:23.159534286 +0000 UTC m=+2286.081992583"
Oct 30 21:48:28.623097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.623051    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"]
Oct 30 21:48:28.628238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.628211    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.646643 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.645585    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"]
Oct 30 21:48:28.652842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.652814    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-db1288-dockercfg-qxc7b\""
Oct 30 21:48:28.653014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.652823    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-db1288\""
Oct 30 21:48:28.719647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.716925    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.719647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.717007    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kbk8\" (UniqueName: \"kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8\") pod \"payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.820351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.818542    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.865653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.861556    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.865653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.861775    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kbk8\" (UniqueName: \"kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8\") pod \"payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.885782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.881948    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"]
Oct 30 21:48:28.889885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.887941    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"]
Oct 30 21:48:28.889885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.888051    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:28.899043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.898724    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kbk8\" (UniqueName: \"kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8\") pod \"payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.927808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.927770    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:28.939495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.939467    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:28.943714 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.943691    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:28.963881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.963857    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:28.964003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:28.963941    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml987\" (UniqueName: \"kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987\") pod \"payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:29.065949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.065715    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:29.065949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.065797    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml987\" (UniqueName: \"kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987\") pod \"payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:29.067268 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.067234    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:29.086136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.086106    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml987\" (UniqueName: \"kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987\") pod \"payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:29.147024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.146940    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:29.220826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.220790    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:29.255259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.255227    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"]
Oct 30 21:48:29.258542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:48:29.258502    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72107bfe_f7a3_4d3c_a808_7ebd99e1d809.slice/crio-a90e18ccd7e4f39d660e3d08db402db24c199a1770055ef48c20061d9abb58a8 WatchSource:0}: Error finding container a90e18ccd7e4f39d660e3d08db402db24c199a1770055ef48c20061d9abb58a8: Status 404 returned error can't find the container with id a90e18ccd7e4f39d660e3d08db402db24c199a1770055ef48c20061d9abb58a8
Oct 30 21:48:29.536857 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:29.536819    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"]
Oct 30 21:48:29.538407 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:48:29.538376    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1396da_8dc6_4dd9_b63e_325dd0e1bf32.slice/crio-02165e63d691ca77e97d362ada8c0b170595ec6ac033cc047a03843e4023b4a4 WatchSource:0}: Error finding container 02165e63d691ca77e97d362ada8c0b170595ec6ac033cc047a03843e4023b4a4: Status 404 returned error can't find the container with id 02165e63d691ca77e97d362ada8c0b170595ec6ac033cc047a03843e4023b4a4
Oct 30 21:48:30.145620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.145580    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" event={"ID":"72107bfe-f7a3-4d3c-a808-7ebd99e1d809","Type":"ContainerStarted","Data":"c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d"}
Oct 30 21:48:30.146144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.145662    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" event={"ID":"72107bfe-f7a3-4d3c-a808-7ebd99e1d809","Type":"ContainerStarted","Data":"a90e18ccd7e4f39d660e3d08db402db24c199a1770055ef48c20061d9abb58a8"}
Oct 30 21:48:30.146144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.145695    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:30.147637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.147596    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr" event={"ID":"da1396da-8dc6-4dd9-b63e-325dd0e1bf32","Type":"ContainerStarted","Data":"4ae576d1ea323bf0c1a9d3077e906cb5658d62ebc54dde8de52d6abf274e74c6"}
Oct 30 21:48:30.147794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.147654    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr" event={"ID":"da1396da-8dc6-4dd9-b63e-325dd0e1bf32","Type":"ContainerStarted","Data":"02165e63d691ca77e97d362ada8c0b170595ec6ac033cc047a03843e4023b4a4"}
Oct 30 21:48:30.147794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.147675    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:30.185008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.184949    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" podStartSLOduration=2.184931835 podStartE2EDuration="2.184931835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:30.182521747 +0000 UTC m=+2293.104980044" watchObservedRunningTime="2026-10-30 21:48:30.184931835 +0000 UTC m=+2293.107390133"
Oct 30 21:48:30.222331 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:30.222264    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr" podStartSLOduration=2.222245403 podStartE2EDuration="2.222245403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:30.222075445 +0000 UTC m=+2293.144533744" watchObservedRunningTime="2026-10-30 21:48:30.222245403 +0000 UTC m=+2293.144703700"
Oct 30 21:48:31.074748 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:48:31.074711    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:48:34.133625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:34.130850    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 21:48:35.102513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:35.102479    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 21:48:41.158977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:41.158935    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 21:48:41.159522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:41.159403    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 21:48:44.025572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.025539    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"]
Oct 30 21:48:44.321133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.321037    6695 generic.go:356] "Generic (PLEG): container finished" podID="8c656625-0f5d-422f-a718-c447c368fcf6" containerID="4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02" exitCode=0
Oct 30 21:48:44.321133 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.321114    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" event={"ID":"8c656625-0f5d-422f-a718-c447c368fcf6","Type":"ContainerDied","Data":"4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"}
Oct 30 21:48:44.321701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.321673    6695 scope.go:122] "RemoveContainer" containerID="4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"
Oct 30 21:48:44.323820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.323794    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:44.323905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.323865    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:44.323905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.323897    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:44.334903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.334872    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"]
Oct 30 21:48:44.339898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.339872    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.357075 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.357054    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:48:44.361683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.361658    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:48:44.362488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.362451    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:48:44.365769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.365747    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"]
Oct 30 21:48:44.457547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.457517    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:48:44.515031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.514988    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.515213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.515120    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9vkf\" (UniqueName: \"kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.515258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.515238    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.616728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.616636    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.616920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.616771    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.616920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.616834    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9vkf\" (UniqueName: \"kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.617683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.617652    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.619226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.619198    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.636704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.636668    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9vkf\" (UniqueName: \"kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf\") pod \"maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:44.684638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:44.684581    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:45.023249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:45.023223    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"]
Oct 30 21:48:45.026027 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:48:45.025986    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc245e48b_5789_48a2_94d4_5b21cc76816a.slice/crio-25448ced2cf32b700a532657621a8d852049380ab0186e3c86e0037828f18d57 WatchSource:0}: Error finding container 25448ced2cf32b700a532657621a8d852049380ab0186e3c86e0037828f18d57: Status 404 returned error can't find the container with id 25448ced2cf32b700a532657621a8d852049380ab0186e3c86e0037828f18d57
Oct 30 21:48:45.362713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:45.362616    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t" event={"ID":"c245e48b-5789-48a2-94d4-5b21cc76816a","Type":"ContainerStarted","Data":"25448ced2cf32b700a532657621a8d852049380ab0186e3c86e0037828f18d57"}
Oct 30 21:48:45.364994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:45.364952    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" event={"ID":"8c656625-0f5d-422f-a718-c447c368fcf6","Type":"ContainerStarted","Data":"80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129"}
Oct 30 21:48:45.585755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:45.585719    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:48:45.586018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:45.585995    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:48:45.586119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:45.586050    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Oct 30 21:48:46.375234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:46.375198    6695 generic.go:356] "Generic (PLEG): container finished" podID="c245e48b-5789-48a2-94d4-5b21cc76816a" containerID="9fb96343730b5af210914cb6956b09b3065c282cd309ed80117d7bdd226ad6fc" exitCode=0
Oct 30 21:48:46.375680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:46.375295    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t" event={"ID":"c245e48b-5789-48a2-94d4-5b21cc76816a","Type":"ContainerDied","Data":"9fb96343730b5af210914cb6956b09b3065c282cd309ed80117d7bdd226ad6fc"}
Oct 30 21:48:46.586187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:46.586144    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:48:46.586375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:46.586213    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Oct 30 21:48:47.585901 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.585862    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:48:47.586387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.585922    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Oct 30 21:48:47.661596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.661569    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:47.827626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.826633    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf\" (UniqueName: \"kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf\") pod \"c245e48b-5789-48a2-94d4-5b21cc76816a\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") "
Oct 30 21:48:47.827626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.826730    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca\") pod \"c245e48b-5789-48a2-94d4-5b21cc76816a\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") "
Oct 30 21:48:47.827626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.826923    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca\") pod \"c245e48b-5789-48a2-94d4-5b21cc76816a\" (UID: \"c245e48b-5789-48a2-94d4-5b21cc76816a\") "
Oct 30 21:48:47.834108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.834065    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca" pod "c245e48b-5789-48a2-94d4-5b21cc76816a" (UID: "c245e48b-5789-48a2-94d4-5b21cc76816a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:47.882242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.880860    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca" pod "c245e48b-5789-48a2-94d4-5b21cc76816a" (UID: "c245e48b-5789-48a2-94d4-5b21cc76816a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:47.891270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.891187    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf" pod "c245e48b-5789-48a2-94d4-5b21cc76816a" (UID: "c245e48b-5789-48a2-94d4-5b21cc76816a"). InnerVolumeSpecName "kube-api-access-z9vkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:47.936635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.930758    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:47.936635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.930799    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9vkf\" (UniqueName: \"kubernetes.io/projected/c245e48b-5789-48a2-94d4-5b21cc76816a-kube-api-access-z9vkf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:47.936635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:47.930816    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c245e48b-5789-48a2-94d4-5b21cc76816a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:48.392232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:48.392202    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"
Oct 30 21:48:48.392524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:48.392239    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t" event={"ID":"c245e48b-5789-48a2-94d4-5b21cc76816a","Type":"ContainerDied","Data":"25448ced2cf32b700a532657621a8d852049380ab0186e3c86e0037828f18d57"}
Oct 30 21:48:48.392524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:48.392296    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25448ced2cf32b700a532657621a8d852049380ab0186e3c86e0037828f18d57"
Oct 30 21:48:48.586227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:48.586195    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:48:48.586649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:48.586253    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Oct 30 21:48:49.586191 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:49.586146    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:48:49.587206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:49.587107    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Oct 30 21:48:50.586704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:50.586660    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:48:50.586928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:50.586737    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Oct 30 21:48:51.591759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:51.591731    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:48:51.592242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:51.592222    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:48:51.592984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:51.592959    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 21:48:55.256651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:55.256612    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"]
Oct 30 21:48:59.047432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.047389    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" podUID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" containerName="main" containerID="cri-o://201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1" gracePeriod=45
Oct 30 21:48:59.136419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.136377    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" podUID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.62:8000/ready\": dial tcp 10.134.1.62:8000: connect: connection refused"
Oct 30 21:48:59.408798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.408746    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:59.499086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.499053    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs\") pod \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") "
Oct 30 21:48:59.499292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.499243    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home\") pod \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") "
Oct 30 21:48:59.499950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.499304    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm\" (UniqueName: \"kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm\") pod \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") "
Oct 30 21:48:59.499950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.499474    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir\") pod \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") "
Oct 30 21:48:59.499950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.499520    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm\") pod \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") "
Oct 30 21:48:59.499950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.499557    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache\") pod \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\" (UID: \"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5\") "
Oct 30 21:48:59.500253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.500095    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home" pod "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" (UID: "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:59.500317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.500250    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:59.500383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.500336    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache" pod "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" (UID: "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:59.500835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.500518    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir" pod "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" (UID: "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:59.502676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.502641    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm" pod "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" (UID: "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:59.503272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.503232    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm" pod "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" (UID: "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5"). InnerVolumeSpecName "kube-api-access-dqfdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:59.503949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.503926    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs" pod "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" (UID: "fb4c6cc2-7690-43bb-82e2-52e36f1d14c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:59.509547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.509519    6695 generic.go:356] "Generic (PLEG): container finished" podID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" containerID="201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1" exitCode=0
Oct 30 21:48:59.509782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.509658    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" event={"ID":"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5","Type":"ContainerDied","Data":"201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1"}
Oct 30 21:48:59.509782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.509697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt" event={"ID":"fb4c6cc2-7690-43bb-82e2-52e36f1d14c5","Type":"ContainerDied","Data":"4f6470ea64423f78c5a54176142a04a4fdeb0529ac91c55e119d448a49ccfacf"}
Oct 30 21:48:59.509782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.509722    6695 scope.go:122] "RemoveContainer" containerID="201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1"
Oct 30 21:48:59.509782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.509762    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"
Oct 30 21:48:59.540310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.540284    6695 scope.go:122] "RemoveContainer" containerID="201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1"
Oct 30 21:48:59.540788 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:48:59.540762    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1\": container with ID starting with 201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1 not found: ID does not exist" containerID="201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1"
Oct 30 21:48:59.540905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.540796    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1"} err="failed to get container status \"201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1\": rpc error: code = NotFound desc = could not find container \"201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1\": container with ID starting with 201c1ac25ed0903f88c6fd14035966f0bf56b537e5a26be955c893d6fe255af1 not found: ID does not exist"
Oct 30 21:48:59.635944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.635857    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqfdm\" (UniqueName: \"kubernetes.io/projected/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-kube-api-access-dqfdm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:59.637919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.637877    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:59.642141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.641012    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"]
Oct 30 21:48:59.644113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.644091    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:59.644491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.644479    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:59.644595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.644585    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:48:59.650814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:48:59.650790    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3bc8ea/autores-model-3bc8ea-kserve-5bf447457c-xtnnt"]
Oct 30 21:49:00.257504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.257394    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv" podUID="e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" containerName="payload-pre-processing" containerID="cri-o://41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665" gracePeriod=25
Oct 30 21:49:00.502554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.502513    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:00.502776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.502618    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:00.667518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.667106    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:49:00.670138 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.670096    6695 generic.go:356] "Generic (PLEG): container finished" podID="e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" containerID="41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665" exitCode=0
Oct 30 21:49:00.670322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.670191    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv" event={"ID":"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa","Type":"ContainerDied","Data":"41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665"}
Oct 30 21:49:00.680691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.671069    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv" event={"ID":"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa","Type":"ContainerDied","Data":"473aac1a1e18ab8a4f040f0cbd73818dbb4bfc8813ed92a6255f710d7492f658"}
Oct 30 21:49:00.680691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.671162    6695 scope.go:122] "RemoveContainer" containerID="41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665"
Oct 30 21:49:00.680691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.674004    6695 generic.go:356] "Generic (PLEG): container finished" podID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerID="921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670" exitCode=0
Oct 30 21:49:00.680691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.674257    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" event={"ID":"bb6d8f36-f381-4ebd-a5a7-912a93c49842","Type":"ContainerDied","Data":"921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"}
Oct 30 21:49:00.680691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.674797    6695 scope.go:122] "RemoveContainer" containerID="921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"
Oct 30 21:49:00.692113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.692046    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:00.692281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.692154    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:00.692281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.692204    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:00.864639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.856291    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd\" (UniqueName: \"kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd\") pod \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") "
Oct 30 21:49:00.871666 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.865154    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume\") pod \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\" (UID: \"e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa\") "
Oct 30 21:49:00.906840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.878522    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume" pod "e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" (UID: "e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:00.919108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.907406    6695 scope.go:122] "RemoveContainer" containerID="41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665"
Oct 30 21:49:00.919542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.907570    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd" pod "e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" (UID: "e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa"). InnerVolumeSpecName "kube-api-access-lzvrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:00.919689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.909995    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:00.925573 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:49:00.924330    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665\": container with ID starting with 41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665 not found: ID does not exist" containerID="41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665"
Oct 30 21:49:00.925573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:00.924377    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665"} err="failed to get container status \"41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665\": rpc error: code = NotFound desc = could not find container \"41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665\": container with ID starting with 41184e05a74e999b861c99c566e2a1c7fb1398d86426c440aa0451e407f85665 not found: ID does not exist"
Oct 30 21:49:01.021578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:01.021536    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvrd\" (UniqueName: \"kubernetes.io/projected/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa-kube-api-access-lzvrd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:01.087040 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:49:01.086953    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:49:01.216550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:01.216510    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" path="/var/lib/kubelet/pods/fb4c6cc2-7690-43bb-82e2-52e36f1d14c5/volumes"
Oct 30 21:49:01.828515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:01.828483    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"
Oct 30 21:49:02.058523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.052697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" event={"ID":"bb6d8f36-f381-4ebd-a5a7-912a93c49842","Type":"ContainerStarted","Data":"91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc"}
Oct 30 21:49:02.066853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.066787    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"]
Oct 30 21:49:02.072651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.072596    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3bc8ea-68f766c459-l75hv"]
Oct 30 21:49:02.143827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.143720    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"]
Oct 30 21:49:02.144493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144474    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c245e48b-5789-48a2-94d4-5b21cc76816a" containerName="revoke-keys"
Oct 30 21:49:02.144560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144495    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c245e48b-5789-48a2-94d4-5b21cc76816a" containerName="revoke-keys"
Oct 30 21:49:02.144560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144532    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" containerName="main"
Oct 30 21:49:02.144560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144537    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" containerName="main"
Oct 30 21:49:02.144740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144565    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" containerName="payload-pre-processing"
Oct 30 21:49:02.144740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144571    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" containerName="payload-pre-processing"
Oct 30 21:49:02.144740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144729    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" containerName="payload-pre-processing"
Oct 30 21:49:02.144740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144747    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c245e48b-5789-48a2-94d4-5b21cc76816a" containerName="revoke-keys"
Oct 30 21:49:02.145669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.144768    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4c6cc2-7690-43bb-82e2-52e36f1d14c5" containerName="main"
Oct 30 21:49:02.149768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.149740    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"]
Oct 30 21:49:02.149895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.149849    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.165016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.164982    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"kube-root-ca.crt\""
Oct 30 21:49:02.165202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.165075    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-c3e67a6d1b10\""
Oct 30 21:49:02.165202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.165189    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-e2e--f95d90a5-dockercfg-67gdj\""
Oct 30 21:49:02.165343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.165274    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"routing-overlay\""
Oct 30 21:49:02.165343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.165336    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-plugins-e2e-extmod-b-846656\""
Oct 30 21:49:02.165466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.165394    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"openshift-service-ca.crt\""
Oct 30 21:49:02.196024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.195980    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:49:02.196237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.196214    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:02.196315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.196272    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:02.200685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.200655    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzb8\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.200865 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.200827    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.200966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.200944    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.201051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.201037    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.302995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.302953    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.303210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.303083    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.303210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.303194    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.303335 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.303242    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzb8\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.304513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.304454    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.306159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.306127    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.330628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.323104    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.360628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.356749    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzb8\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.471681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.470866    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:02.844921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:02.844874    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"]
Oct 30 21:49:02.849064 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:02.849018    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6b97e_802e_4445_a061_f946b78fa24e.slice/crio-258037725bf848580a35a1c98c1befe4eca46d6b01121e8d08962fad788a8d70 WatchSource:0}: Error finding container 258037725bf848580a35a1c98c1befe4eca46d6b01121e8d08962fad788a8d70: Status 404 returned error can't find the container with id 258037725bf848580a35a1c98c1befe4eca46d6b01121e8d08962fad788a8d70
Oct 30 21:49:03.064024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.063984    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz" event={"ID":"e0c6b97e-802e-4445-a061-f946b78fa24e","Type":"ContainerStarted","Data":"3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669"}
Oct 30 21:49:03.064211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.064047    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz" event={"ID":"e0c6b97e-802e-4445-a061-f946b78fa24e","Type":"ContainerStarted","Data":"258037725bf848580a35a1c98c1befe4eca46d6b01121e8d08962fad788a8d70"}
Oct 30 21:49:03.064287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.064267    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:03.105684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.105544    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz" podStartSLOduration=1.105523435 podStartE2EDuration="1.105523435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:03.102591268 +0000 UTC m=+2326.025049580" watchObservedRunningTime="2026-10-30 21:49:03.105523435 +0000 UTC m=+2326.027981734"
Oct 30 21:49:03.196171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.196134    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:03.196398 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.196210    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:03.292273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:03.292233    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa" path="/var/lib/kubelet/pods/e70ba5b3-fcb2-4334-91ff-f2b7b1ab11aa/volumes"
Oct 30 21:49:04.196505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:04.196469    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:04.196923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:04.196528    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:05.196037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.196000    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:05.196236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.196057    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:05.553040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.552995    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"]
Oct 30 21:49:05.557968 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.557935    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.593194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.579691    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9722ac-openshift-default-dockercfg-d87s2\""
Oct 30 21:49:05.596429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.596392    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"]
Oct 30 21:49:05.652720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.652675    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.652948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.652827    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.652948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.652896    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.653050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.652964    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrhc\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.653180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.653151    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.653236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.653202    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.653309 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.653293    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.653357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.653333    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.653499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.653479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755228    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755256    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755288    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrhc\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755425    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755457    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755495    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.755616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755534    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.756342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.756316    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.756542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.756523    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.757792 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.757765    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.758050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.758029    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.758555 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.758530    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.768503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.765969    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.768503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.755647    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.800626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.786143    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrhc\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.800626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.800315    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.807726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.807154    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token\") pod \"e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") " pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:05.888640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:05.877853    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:06.198193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:06.198154    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:06.198369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:06.198218    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:06.397568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:06.392465    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"]
Oct 30 21:49:06.458778 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:06.458712    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406109b5_5975_4c91_95e7_739fa653abbb.slice/crio-3c11752dbda83a3ba344865740e320b308457f8ff3e1d91a4124d0d1157e56c7 WatchSource:0}: Error finding container 3c11752dbda83a3ba344865740e320b308457f8ff3e1d91a4124d0d1157e56c7: Status 404 returned error can't find the container with id 3c11752dbda83a3ba344865740e320b308457f8ff3e1d91a4124d0d1157e56c7
Oct 30 21:49:06.476629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:06.462776    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:06.476629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:06.462883    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:06.476629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:06.462927    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:07.102614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.102574    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" event={"ID":"406109b5-5975-4c91-95e7-739fa653abbb","Type":"ContainerStarted","Data":"0d0271f20c5e60e937f781321e88ab1813cc84cda59f70822005e040b309b9e7"}
Oct 30 21:49:07.103045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.102651    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" event={"ID":"406109b5-5975-4c91-95e7-739fa653abbb","Type":"ContainerStarted","Data":"3c11752dbda83a3ba344865740e320b308457f8ff3e1d91a4124d0d1157e56c7"}
Oct 30 21:49:07.165033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.164961    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podStartSLOduration=2.164932629 podStartE2EDuration="2.164932629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:07.15972479 +0000 UTC m=+2330.082183088" watchObservedRunningTime="2026-10-30 21:49:07.164932629 +0000 UTC m=+2330.087390930"
Oct 30 21:49:07.196378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.196343    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:49:07.196571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.196398    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.60:15021/healthz/ready\": dial tcp 10.134.1.60:15021: connect: connection refused"
Oct 30 21:49:07.879500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.879446    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:07.887739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:07.887705    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:08.138540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:08.138454    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:08.155308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:08.155276    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:49:08.202170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:08.202134    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:49:08.202556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:08.202532    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:49:08.203485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:08.203463    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 21:49:11.896104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.896051    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"]
Oct 30 21:49:11.901072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.901051    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:11.918918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.918889    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"]
Oct 30 21:49:11.926077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.926049    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9722ac\""
Oct 30 21:49:11.963101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.963060    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"]
Oct 30 21:49:11.969038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.969013    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:11.983287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.983253    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9722ac-dockercfg-qbfq7\""
Oct 30 21:49:11.986400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.986374    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"]
Oct 30 21:49:11.995778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.995740    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8tmc\" (UniqueName: \"kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc\") pod \"payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:11.995939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.995885    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:11.996068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.996046    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrvh\" (UniqueName: \"kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh\") pod \"payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:11.996177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:11.996160    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:12.109624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.107532    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:12.111790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.111746    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8tmc\" (UniqueName: \"kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc\") pod \"payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:12.111933 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.111853    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:12.112044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.112027    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrvh\" (UniqueName: \"kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh\") pod \"payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:12.121456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.113340    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:12.121456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.114385    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:12.176915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.176678    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrvh\" (UniqueName: \"kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh\") pod \"payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:12.209855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.209813    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8tmc\" (UniqueName: \"kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc\") pod \"payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:12.219627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.217260    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:12.296621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.295227    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:12.542930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.542894    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"]
Oct 30 21:49:12.546523 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:12.546492    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9425b836_96bc_4f48_a6a1_5370bbc5ab0f.slice/crio-6981779eb80c6cbf5e136cba73aa70eeb4edd9ce246fedd50445a370631eb2f6 WatchSource:0}: Error finding container 6981779eb80c6cbf5e136cba73aa70eeb4edd9ce246fedd50445a370631eb2f6: Status 404 returned error can't find the container with id 6981779eb80c6cbf5e136cba73aa70eeb4edd9ce246fedd50445a370631eb2f6
Oct 30 21:49:12.640663 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:12.640597    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3307ce_1d41_45c8_b4a3_fd9bbc750813.slice/crio-bbcfa3fe3b289ad64feb99436679aa24cc82828a6d8fa86979d32fba916f3f21 WatchSource:0}: Error finding container bbcfa3fe3b289ad64feb99436679aa24cc82828a6d8fa86979d32fba916f3f21: Status 404 returned error can't find the container with id bbcfa3fe3b289ad64feb99436679aa24cc82828a6d8fa86979d32fba916f3f21
Oct 30 21:49:12.641273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:12.641246    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"]
Oct 30 21:49:13.236375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.236337    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" event={"ID":"9425b836-96bc-4f48-a6a1-5370bbc5ab0f","Type":"ContainerStarted","Data":"80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91"}
Oct 30 21:49:13.236842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.236391    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" event={"ID":"9425b836-96bc-4f48-a6a1-5370bbc5ab0f","Type":"ContainerStarted","Data":"6981779eb80c6cbf5e136cba73aa70eeb4edd9ce246fedd50445a370631eb2f6"}
Oct 30 21:49:13.236842 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.236415    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:13.238227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.238203    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" event={"ID":"2e3307ce-1d41-45c8-b4a3-fd9bbc750813","Type":"ContainerStarted","Data":"f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48"}
Oct 30 21:49:13.238330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.238246    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" event={"ID":"2e3307ce-1d41-45c8-b4a3-fd9bbc750813","Type":"ContainerStarted","Data":"bbcfa3fe3b289ad64feb99436679aa24cc82828a6d8fa86979d32fba916f3f21"}
Oct 30 21:49:13.238330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.238284    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:13.273389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.273335    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" podStartSLOduration=2.273322602 podStartE2EDuration="2.273322602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:13.272437023 +0000 UTC m=+2336.194895320" watchObservedRunningTime="2026-10-30 21:49:13.273322602 +0000 UTC m=+2336.195780899"
Oct 30 21:49:13.310813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:13.310750    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" podStartSLOduration=2.310733141 podStartE2EDuration="2.310733141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:13.309350285 +0000 UTC m=+2336.231808595" watchObservedRunningTime="2026-10-30 21:49:13.310733141 +0000 UTC m=+2336.233191439"
Oct 30 21:49:14.075708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:14.075676    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:49:22.146210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:22.146173    6695 scope.go:122] "RemoveContainer" containerID="615797fb7170449c0c048455df82e54f5b84654f3f2fb3e47a8f918177102fc8"
Oct 30 21:49:24.249408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:24.249370    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 21:49:24.249838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:24.249445    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 21:49:24.918118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:24.918082    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"]
Oct 30 21:49:24.929089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:24.929053    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3bc8ea-6ml5t"]
Oct 30 21:49:25.215776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:25.215694    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c245e48b-5789-48a2-94d4-5b21cc76816a" path="/var/lib/kubelet/pods/c245e48b-5789-48a2-94d4-5b21cc76816a/volumes"
Oct 30 21:49:31.074793 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:49:31.074714    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:49:31.806306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:31.806265    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"]
Oct 30 21:49:31.806567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:31.806535    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy" containerID="cri-o://ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751" gracePeriod=30
Oct 30 21:49:37.120277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.120242    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232041    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232170    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232243    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232274    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232334    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232426    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232501    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232536    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.233713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.232673    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6\") pod \"1552f927-4527-4263-bcea-e543aeb57e73\" (UID: \"1552f927-4527-4263-bcea-e543aeb57e73\") "
Oct 30 21:49:37.253235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.252919    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:37.292875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.262969    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:37.292875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.263813    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:37.292875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.264103    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:37.292875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.264390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:37.320723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.315654    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:49:37.320723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.316378    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "kube-api-access-cfcb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:37.320723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.316457    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:37.320723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.316512    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy" pod "1552f927-4527-4263-bcea-e543aeb57e73" (UID: "1552f927-4527-4263-bcea-e543aeb57e73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336807    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1552f927-4527-4263-bcea-e543aeb57e73-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336842    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336865    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336881    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336896    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336909    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336923    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1552f927-4527-4263-bcea-e543aeb57e73-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336938    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcb6\" (UniqueName: \"kubernetes.io/projected/1552f927-4527-4263-bcea-e543aeb57e73-kube-api-access-cfcb6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.342414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.336952    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1552f927-4527-4263-bcea-e543aeb57e73-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:49:37.471760 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.471725    6695 generic.go:356] "Generic (PLEG): container finished" podID="1552f927-4527-4263-bcea-e543aeb57e73" containerID="ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751" exitCode=0
Oct 30 21:49:37.472025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.472003    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" event={"ID":"1552f927-4527-4263-bcea-e543aeb57e73","Type":"ContainerDied","Data":"ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751"}
Oct 30 21:49:37.472115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.472066    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb" event={"ID":"1552f927-4527-4263-bcea-e543aeb57e73","Type":"ContainerDied","Data":"1b49f361ce6a75638fdf773725a904a42bcf4095af4b108d45422f3d78fc3d72"}
Oct 30 21:49:37.472115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.472102    6695 scope.go:122] "RemoveContainer" containerID="ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751"
Oct 30 21:49:37.472281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.472265    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"
Oct 30 21:49:37.488146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.488119    6695 scope.go:122] "RemoveContainer" containerID="64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"
Oct 30 21:49:37.506040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.506013    6695 scope.go:122] "RemoveContainer" containerID="ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751"
Oct 30 21:49:37.506467 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:49:37.506435    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751\": container with ID starting with ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751 not found: ID does not exist" containerID="ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751"
Oct 30 21:49:37.506677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.506480    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751"} err="failed to get container status \"ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751\": rpc error: code = NotFound desc = could not find container \"ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751\": container with ID starting with ec6a89588a1a4b0dd6cc78211da0502298215d703db09e0aa44a37a623742751 not found: ID does not exist"
Oct 30 21:49:37.506677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.506504    6695 scope.go:122] "RemoveContainer" containerID="64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"
Oct 30 21:49:37.507289 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:49:37.507255    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3\": container with ID starting with 64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3 not found: ID does not exist" containerID="64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"
Oct 30 21:49:37.507387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.507297    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3"} err="failed to get container status \"64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3\": rpc error: code = NotFound desc = could not find container \"64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3\": container with ID starting with 64ad38f2db1c13ec013e85f02f40bf2e8f53495dbef76ea20b237a8e4e2d72b3 not found: ID does not exist"
Oct 30 21:49:37.544295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.544238    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"]
Oct 30 21:49:37.546578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:37.546540    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3bc8ea-openshift-default-547db665f9-9jksb"]
Oct 30 21:49:39.228995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:39.228962    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1552f927-4527-4263-bcea-e543aeb57e73" path="/var/lib/kubelet/pods/1552f927-4527-4263-bcea-e543aeb57e73/volumes"
Oct 30 21:49:40.043938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.043900    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"]
Oct 30 21:49:40.044762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.044739    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy"
Oct 30 21:49:40.044897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.044762    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy"
Oct 30 21:49:40.044897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.044853    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy"
Oct 30 21:49:40.044897 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.044862    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy"
Oct 30 21:49:40.045086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.045061    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy"
Oct 30 21:49:40.050697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.050671    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.071893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.071858    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5d3601\""
Oct 30 21:49:40.074369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.074335    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"]
Oct 30 21:49:40.081567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.081544    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5d3601-dockercfg-q92qb\""
Oct 30 21:49:40.148438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.148403    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"]
Oct 30 21:49:40.150194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.150159    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1552f927-4527-4263-bcea-e543aeb57e73" containerName="istio-proxy"
Oct 30 21:49:40.179628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.178479    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.213978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.210740    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.213978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.210951    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbkw\" (UniqueName: \"kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw\") pod \"payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.312981 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.312878    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.313621 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.313574    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn8q\" (UniqueName: \"kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q\") pod \"payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.314036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.314007    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbkw\" (UniqueName: \"kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw\") pod \"payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.314499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.314461    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.316067 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.316029    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.323886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.323856    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"]
Oct 30 21:49:40.345175 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.345144    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbkw\" (UniqueName: \"kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw\") pod \"payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.365271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.365226    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:40.416244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.416202    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.416460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.416377    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn8q\" (UniqueName: \"kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q\") pod \"payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.417013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.416986    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.450023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.449954    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn8q\" (UniqueName: \"kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q\") pod \"payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.635653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.635561    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:40.678096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.678067    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"]
Oct 30 21:49:40.681542 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:40.681508    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99522c1_9a6c_4985_bf28_1a45619b9dd5.slice/crio-dff843568f37268f5a01a8c0a579bed4e07bb3a86a3c3bcef0f630b7bd69015f WatchSource:0}: Error finding container dff843568f37268f5a01a8c0a579bed4e07bb3a86a3c3bcef0f630b7bd69015f: Status 404 returned error can't find the container with id dff843568f37268f5a01a8c0a579bed4e07bb3a86a3c3bcef0f630b7bd69015f
Oct 30 21:49:40.947340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:40.947313    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"]
Oct 30 21:49:40.949789 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:40.949762    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebf10d2_094f_4568_8f42_1c15cb8b786b.slice/crio-72f11867ea751a04febad6f662adac873b0817a39597f18879df924f79562c7e WatchSource:0}: Error finding container 72f11867ea751a04febad6f662adac873b0817a39597f18879df924f79562c7e: Status 404 returned error can't find the container with id 72f11867ea751a04febad6f662adac873b0817a39597f18879df924f79562c7e
Oct 30 21:49:41.506258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.506210    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" event={"ID":"a99522c1-9a6c-4985-bf28-1a45619b9dd5","Type":"ContainerStarted","Data":"a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6"}
Oct 30 21:49:41.506790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.506283    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:41.506790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.506308    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" event={"ID":"a99522c1-9a6c-4985-bf28-1a45619b9dd5","Type":"ContainerStarted","Data":"dff843568f37268f5a01a8c0a579bed4e07bb3a86a3c3bcef0f630b7bd69015f"}
Oct 30 21:49:41.508152 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.508120    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" event={"ID":"bebf10d2-094f-4568-8f42-1c15cb8b786b","Type":"ContainerStarted","Data":"e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31"}
Oct 30 21:49:41.508282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.508187    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:41.508282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.508211    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" event={"ID":"bebf10d2-094f-4568-8f42-1c15cb8b786b","Type":"ContainerStarted","Data":"72f11867ea751a04febad6f662adac873b0817a39597f18879df924f79562c7e"}
Oct 30 21:49:41.604767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.604696    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" podStartSLOduration=1.6046806089999999 podStartE2EDuration="1.604680609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:41.602975162 +0000 UTC m=+2364.525433473" watchObservedRunningTime="2026-10-30 21:49:41.604680609 +0000 UTC m=+2364.527138907"
Oct 30 21:49:41.607214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:41.607168    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" podStartSLOduration=1.6071575500000002 podStartE2EDuration="1.60715755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:41.557348075 +0000 UTC m=+2364.479806374" watchObservedRunningTime="2026-10-30 21:49:41.60715755 +0000 UTC m=+2364.529615894"
Oct 30 21:49:52.518932 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:52.518897    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 21:49:52.519450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:52.519292    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 21:49:53.015707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.015668    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"]
Oct 30 21:49:53.021151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.021122    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.039186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.039154    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"]
Oct 30 21:49:53.041212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.041185    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.041332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.041277    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25vv\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.041517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.041496    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.041568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.041529    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.046167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.046037    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"openshift-service-ca.crt\""
Oct 30 21:49:53.046167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.046058    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-plugins-e2e-extmod-b-db1288\""
Oct 30 21:49:53.046167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.046037    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-c26bbe9ee934\""
Oct 30 21:49:53.046167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.046083    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"kube-root-ca.crt\""
Oct 30 21:49:53.046167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.046059    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"routing-overlay\""
Oct 30 21:49:53.046528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.046292    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-e2e--f0c0883f-dockercfg-8mgn7\""
Oct 30 21:49:53.149631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.143673    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.149631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.143790    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.149631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.143856    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25vv\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.149631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.144134    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.149631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.145260    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.188624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.174678    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.207247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.207188    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.238641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.224423    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25vv\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.337572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.337533    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:53.657963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:53.657937    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"]
Oct 30 21:49:53.661366 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:49:53.661335    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934f90c1_18ae_4376_a57f_eaf17d039e38.slice/crio-a24eb355c19b77e528f38a0a06f011165459e03e3e877cee2d119c787cce64b9 WatchSource:0}: Error finding container a24eb355c19b77e528f38a0a06f011165459e03e3e877cee2d119c787cce64b9: Status 404 returned error can't find the container with id a24eb355c19b77e528f38a0a06f011165459e03e3e877cee2d119c787cce64b9
Oct 30 21:49:54.669785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:54.669746    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz" event={"ID":"934f90c1-18ae-4376-a57f-eaf17d039e38","Type":"ContainerStarted","Data":"2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0"}
Oct 30 21:49:54.670256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:54.669800    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz" event={"ID":"934f90c1-18ae-4376-a57f-eaf17d039e38","Type":"ContainerStarted","Data":"a24eb355c19b77e528f38a0a06f011165459e03e3e877cee2d119c787cce64b9"}
Oct 30 21:49:54.670256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:54.669965    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:49:54.712770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:49:54.712709    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz" podStartSLOduration=2.712693629 podStartE2EDuration="2.712693629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:54.707741413 +0000 UTC m=+2377.630199700" watchObservedRunningTime="2026-10-30 21:49:54.712693629 +0000 UTC m=+2377.635151927"
Oct 30 21:50:01.078945 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:50:01.078907    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:50:05.682019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:05.681988    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:50:18.936336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:18.936300    6695 generic.go:356] "Generic (PLEG): container finished" podID="406109b5-5975-4c91-95e7-739fa653abbb" containerID="0d0271f20c5e60e937f781321e88ab1813cc84cda59f70822005e040b309b9e7" exitCode=0
Oct 30 21:50:18.936898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:18.936371    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" event={"ID":"406109b5-5975-4c91-95e7-739fa653abbb","Type":"ContainerDied","Data":"0d0271f20c5e60e937f781321e88ab1813cc84cda59f70822005e040b309b9e7"}
Oct 30 21:50:18.936951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:18.936925    6695 scope.go:122] "RemoveContainer" containerID="0d0271f20c5e60e937f781321e88ab1813cc84cda59f70822005e040b309b9e7"
Oct 30 21:50:18.938913 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:18.938885    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:50:18.939010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:18.938957    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:50:18.939010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:18.938991    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:50:19.986021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:19.985956    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" event={"ID":"406109b5-5975-4c91-95e7-739fa653abbb","Type":"ContainerStarted","Data":"14e4bbab468d541eb545ea9379d4bddb6df46270ddf02285f7c1623e821f65e3"}
Oct 30 21:50:20.015140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:20.015104    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:50:20.048451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:20.048423    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:50:20.878505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:20.878460    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:50:20.879875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:20.879854    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body=
Oct 30 21:50:20.879967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:20.879898    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused"
Oct 30 21:50:21.878575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:21.878537    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body=
Oct 30 21:50:21.878983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:21.878618    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused"
Oct 30 21:50:22.188248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:22.188165    6695 scope.go:122] "RemoveContainer" containerID="688c4da4de7ed1493e06585fb816073021fd6e16cf131d66a799fc4743760117"
Oct 30 21:50:22.879235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:22.879197    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body=
Oct 30 21:50:22.879674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:22.879265    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused"
Oct 30 21:50:23.138400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:23.138296    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:50:23.879426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:23.879390    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body=
Oct 30 21:50:23.879870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:23.879455    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused"
Oct 30 21:50:24.879147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:24.879107    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body=
Oct 30 21:50:24.879376 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:24.879184    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused"
Oct 30 21:50:25.884027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:25.883998    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:50:25.885193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:25.885168    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 21:50:31.075250 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:50:31.075210    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:50:31.555617 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.555565    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"]
Oct 30 21:50:31.560065 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.560043    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.585059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.585015    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-234904d763c3\""
Oct 30 21:50:31.589727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.589695    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"]
Oct 30 21:50:31.601562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.601526    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wz7d\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.601821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.601630    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.601919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.601898    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.602011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.601941    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.703547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.703505    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.703547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.703555    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.703821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.703647    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wz7d\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.703821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.703717    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.704326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.704303    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.704382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.704355    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.706200 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.706182    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.722203 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.722179    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wz7d\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:31.874150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:31.874063    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:32.184017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:32.183985    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"]
Oct 30 21:50:32.187358 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:50:32.187328    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71caf68f_87d4_4f22_be9b_4bb1cda97805.slice/crio-f3baa366c0048a507946ea875aa987a5299a1c7711084c2912d09215b3e89bf6 WatchSource:0}: Error finding container f3baa366c0048a507946ea875aa987a5299a1c7711084c2912d09215b3e89bf6: Status 404 returned error can't find the container with id f3baa366c0048a507946ea875aa987a5299a1c7711084c2912d09215b3e89bf6
Oct 30 21:50:33.119638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:33.119580    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn" event={"ID":"71caf68f-87d4-4f22-be9b-4bb1cda97805","Type":"ContainerStarted","Data":"20fd640078f84062e91374f394e586b16b840f1ff81429c9b4d07e7548fdcf0f"}
Oct 30 21:50:33.119862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:33.119653    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn" event={"ID":"71caf68f-87d4-4f22-be9b-4bb1cda97805","Type":"ContainerStarted","Data":"f3baa366c0048a507946ea875aa987a5299a1c7711084c2912d09215b3e89bf6"}
Oct 30 21:50:33.119862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:33.119741    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:33.172838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:33.172777    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn" podStartSLOduration=2.172759917 podStartE2EDuration="2.172759917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:33.166025889 +0000 UTC m=+2416.088484190" watchObservedRunningTime="2026-10-30 21:50:33.172759917 +0000 UTC m=+2416.095218215"
Oct 30 21:50:35.710109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.710071    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"]
Oct 30 21:50:35.713597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.713577    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.733192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.733160    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"]
Oct 30 21:50:35.740204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.740176    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9722ac\"/\"routing-overlay\""
Oct 30 21:50:35.740397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.740180    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9722ac\"/\"payload-processing-external-model-plugins-e2e-extmod-b-9722ac\""
Oct 30 21:50:35.740397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.740194    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9722ac\"/\"openshift-service-ca.crt\""
Oct 30 21:50:35.740552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.740205    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9722ac\"/\"kube-root-ca.crt\""
Oct 30 21:50:35.740552 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.740213    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9722ac\"/\"agc-cred-3dded3b7fd3b\""
Oct 30 21:50:35.742748 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.742722    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9722ac\"/\"payload-processing-external-model-e2e--718f7756-dockercfg-wj6zm\""
Oct 30 21:50:35.810996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.810960    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.811223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.811167    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.811223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.811211    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.811382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.811267    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwlg\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.912584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.912541    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.913966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.913933    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.914725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.914707    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.914888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.914872    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwlg\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.915243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.913703    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.915243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.914646    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.939677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.930463    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:35.974751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:35.964222    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwlg\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg\") pod \"payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") " pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:36.028120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:36.028084    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:36.361105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:36.361076    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"]
Oct 30 21:50:36.363489 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:50:36.363459    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ca39d4_0300_491c_8625_aa0642eec4e1.slice/crio-ad83bc31f12a5d7646d1fe6a289e8136145d682098e477544eb6688169d8c1b9 WatchSource:0}: Error finding container ad83bc31f12a5d7646d1fe6a289e8136145d682098e477544eb6688169d8c1b9: Status 404 returned error can't find the container with id ad83bc31f12a5d7646d1fe6a289e8136145d682098e477544eb6688169d8c1b9
Oct 30 21:50:37.161300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:37.161263    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg" event={"ID":"81ca39d4-0300-491c-8625-aa0642eec4e1","Type":"ContainerStarted","Data":"e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2"}
Oct 30 21:50:37.161844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:37.161325    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg" event={"ID":"81ca39d4-0300-491c-8625-aa0642eec4e1","Type":"ContainerStarted","Data":"ad83bc31f12a5d7646d1fe6a289e8136145d682098e477544eb6688169d8c1b9"}
Oct 30 21:50:37.161844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:37.161351    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:37.202174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:37.202121    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg" podStartSLOduration=2.202105758 podStartE2EDuration="2.202105758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:37.198536453 +0000 UTC m=+2420.120994751" watchObservedRunningTime="2026-10-30 21:50:37.202105758 +0000 UTC m=+2420.124564064"
Oct 30 21:50:41.030084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:41.030046    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"]
Oct 30 21:50:41.041740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:41.041688    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b21fa274-n6lf4"]
Oct 30 21:50:41.215814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:41.215780    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26b965c-48a7-4920-96aa-c7dab4bbfefa" path="/var/lib/kubelet/pods/e26b965c-48a7-4920-96aa-c7dab4bbfefa/volumes"
Oct 30 21:50:44.132457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:44.132417    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:50:44.205443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:44.205409    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"]
Oct 30 21:50:48.171448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:48.171418    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:50:49.205930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.205886    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz" podUID="e0c6b97e-802e-4445-a061-f946b78fa24e" containerName="payload-processing" containerID="cri-o://3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669" gracePeriod=25
Oct 30 21:50:49.493571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.493543    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:50:49.524577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.524535    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume\") pod \"e0c6b97e-802e-4445-a061-f946b78fa24e\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") "
Oct 30 21:50:49.524798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.524727    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8\") pod \"e0c6b97e-802e-4445-a061-f946b78fa24e\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") "
Oct 30 21:50:49.524798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.524787    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume\") pod \"e0c6b97e-802e-4445-a061-f946b78fa24e\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") "
Oct 30 21:50:49.524957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.524886    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials\") pod \"e0c6b97e-802e-4445-a061-f946b78fa24e\" (UID: \"e0c6b97e-802e-4445-a061-f946b78fa24e\") "
Oct 30 21:50:49.525097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.525067    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume" pod "e0c6b97e-802e-4445-a061-f946b78fa24e" (UID: "e0c6b97e-802e-4445-a061-f946b78fa24e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:49.525523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.525413    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume" pod "e0c6b97e-802e-4445-a061-f946b78fa24e" (UID: "e0c6b97e-802e-4445-a061-f946b78fa24e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:49.525669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.525614    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:50:49.525669 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.525639    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b97e-802e-4445-a061-f946b78fa24e-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:50:49.527281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.527260    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials" pod "e0c6b97e-802e-4445-a061-f946b78fa24e" (UID: "e0c6b97e-802e-4445-a061-f946b78fa24e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:49.527401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.527383    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8" pod "e0c6b97e-802e-4445-a061-f946b78fa24e" (UID: "e0c6b97e-802e-4445-a061-f946b78fa24e"). InnerVolumeSpecName "kube-api-access-pdzb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:49.626739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.626702    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdzb8\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-kube-api-access-pdzb8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:50:49.626739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:49.626736    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0c6b97e-802e-4445-a061-f946b78fa24e-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:50:50.288764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.288723    6695 generic.go:356] "Generic (PLEG): container finished" podID="e0c6b97e-802e-4445-a061-f946b78fa24e" containerID="3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669" exitCode=0
Oct 30 21:50:50.289248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.288763    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz" event={"ID":"e0c6b97e-802e-4445-a061-f946b78fa24e","Type":"ContainerDied","Data":"3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669"}
Oct 30 21:50:50.289248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.288785    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"
Oct 30 21:50:50.289248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.288840    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz" event={"ID":"e0c6b97e-802e-4445-a061-f946b78fa24e","Type":"ContainerDied","Data":"258037725bf848580a35a1c98c1befe4eca46d6b01121e8d08962fad788a8d70"}
Oct 30 21:50:50.289248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.288886    6695 scope.go:122] "RemoveContainer" containerID="3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669"
Oct 30 21:50:50.301964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.301939    6695 scope.go:122] "RemoveContainer" containerID="3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669"
Oct 30 21:50:50.302359 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:50:50.302337    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669\": container with ID starting with 3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669 not found: ID does not exist" containerID="3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669"
Oct 30 21:50:50.302438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.302368    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669"} err="failed to get container status \"3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669\": rpc error: code = NotFound desc = could not find container \"3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669\": container with ID starting with 3ef8decc14080aca40266c2f8cd4286d4aaba8662f39c8b5e1e9c1f663dcc669 not found: ID does not exist"
Oct 30 21:50:50.340014 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.339976    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"]
Oct 30 21:50:50.353054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:50.353014    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf54cwz"]
Oct 30 21:50:51.216116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:51.216079    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c6b97e-802e-4445-a061-f946b78fa24e" path="/var/lib/kubelet/pods/e0c6b97e-802e-4445-a061-f946b78fa24e/volumes"
Oct 30 21:50:52.028279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:52.028241    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"]
Oct 30 21:50:52.038670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:52.038628    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e8183a2-9s7vw"]
Oct 30 21:50:53.215859 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:50:53.215828    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431b2737-eddb-4c5a-90a8-7d7fa87b0fb0" path="/var/lib/kubelet/pods/431b2737-eddb-4c5a-90a8-7d7fa87b0fb0/volumes"
Oct 30 21:51:01.026770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.026688    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"]
Oct 30 21:51:01.027562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.027541    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6b97e-802e-4445-a061-f946b78fa24e" containerName="payload-processing"
Oct 30 21:51:01.027562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.027560    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6b97e-802e-4445-a061-f946b78fa24e" containerName="payload-processing"
Oct 30 21:51:01.027835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.027821    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6b97e-802e-4445-a061-f946b78fa24e" containerName="payload-processing"
Oct 30 21:51:01.033369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.033349    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.056522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.056495    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"openshift-service-ca.crt\""
Oct 30 21:51:01.068173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.068092    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"kube-root-ca.crt\""
Oct 30 21:51:01.068173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.068126    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdb168\""
Oct 30 21:51:01.068487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.068244    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"routing-overlay\""
Oct 30 21:51:01.068487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.068263    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"agc-cred-ef6210aa91ed\""
Oct 30 21:51:01.068487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.068262    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"agc-cred-e2da5504c633\""
Oct 30 21:51:01.068487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.068446    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"payload-processing-external-model-e2e--ca9775af-dockercfg-772nl\""
Oct 30 21:51:01.074103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.074068    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"]
Oct 30 21:51:01.082811 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:51:01.082772    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:51:01.089796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.089761    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.089983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.089855    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v574n\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.089983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.089926    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.090089 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.090044    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.191283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.191241    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.191506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.191427    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.191596 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.191512    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v574n\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.191690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.191597    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.192438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.192406    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.192583 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.192463    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.194526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.194502    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.211131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.211100    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v574n\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.350678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.349636    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:01.792668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.792641    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"]
Oct 30 21:51:01.795676 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:01.795642    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc422d8_3ab1_463e_b75a_39216939ab8a.slice/crio-7edbe624862466915e69046eca2d987ccfe40fa2a27529e44ddac23aa8c0e227 WatchSource:0}: Error finding container 7edbe624862466915e69046eca2d987ccfe40fa2a27529e44ddac23aa8c0e227: Status 404 returned error can't find the container with id 7edbe624862466915e69046eca2d987ccfe40fa2a27529e44ddac23aa8c0e227
Oct 30 21:51:01.979829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.979790    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"]
Oct 30 21:51:01.984713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:01.984684    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.001283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.001255    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-661a593484c5\""
Oct 30 21:51:02.008653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.006120    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.008653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.006191    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.008653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.006247    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.008653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.006337    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"]
Oct 30 21:51:02.008653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.006544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvmh\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.119213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.119055    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvmh\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.119213 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.119205    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.119829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.119256    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.119829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.119283    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.120247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.120192    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.120402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.120276    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.122776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.122749    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.137419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.137384    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvmh\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh\") pod \"payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.300028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.299989    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:02.486359 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.486309    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx" event={"ID":"2fc422d8-3ab1-463e-b75a-39216939ab8a","Type":"ContainerStarted","Data":"57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe"}
Oct 30 21:51:02.486540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.486385    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx" event={"ID":"2fc422d8-3ab1-463e-b75a-39216939ab8a","Type":"ContainerStarted","Data":"7edbe624862466915e69046eca2d987ccfe40fa2a27529e44ddac23aa8c0e227"}
Oct 30 21:51:02.486540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.486453    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:02.526183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.526126    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx" podStartSLOduration=2.526107789 podStartE2EDuration="2.526107789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:02.524698601 +0000 UTC m=+2445.447156911" watchObservedRunningTime="2026-10-30 21:51:02.526107789 +0000 UTC m=+2445.448566088"
Oct 30 21:51:02.661529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:02.661456    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"]
Oct 30 21:51:02.664576 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:02.664545    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf291c364_1cea_464b_aad9_9637fdb5d474.slice/crio-53e966c05f286f7d6bccbb276a42ce8efde91a746a3a295823eeb8ebb28136c9 WatchSource:0}: Error finding container 53e966c05f286f7d6bccbb276a42ce8efde91a746a3a295823eeb8ebb28136c9: Status 404 returned error can't find the container with id 53e966c05f286f7d6bccbb276a42ce8efde91a746a3a295823eeb8ebb28136c9
Oct 30 21:51:03.546625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:03.542988    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8" event={"ID":"f291c364-1cea-464b-aad9-9637fdb5d474","Type":"ContainerStarted","Data":"9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513"}
Oct 30 21:51:03.546625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:03.543047    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8" event={"ID":"f291c364-1cea-464b-aad9-9637fdb5d474","Type":"ContainerStarted","Data":"53e966c05f286f7d6bccbb276a42ce8efde91a746a3a295823eeb8ebb28136c9"}
Oct 30 21:51:03.546625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:03.543106    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:03.592738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:03.592677    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8" podStartSLOduration=2.592655171 podStartE2EDuration="2.592655171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:03.585690316 +0000 UTC m=+2446.508148615" watchObservedRunningTime="2026-10-30 21:51:03.592655171 +0000 UTC m=+2446.515113471"
Oct 30 21:51:07.392579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.392540    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"]
Oct 30 21:51:07.397786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.397757    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.421586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.421555    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ed0866-dockercfg-9f8xk\""
Oct 30 21:51:07.421781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.421635    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ed0866\""
Oct 30 21:51:07.434003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.433970    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"]
Oct 30 21:51:07.484776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.484741    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"]
Oct 30 21:51:07.490282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.490256    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:07.509306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.509266    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"]
Oct 30 21:51:07.721925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.721825    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.721925 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.721873    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5k4p\" (UniqueName: \"kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p\") pod \"payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.722189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.722099    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:07.722265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.722216    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms\") pod \"payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:07.823327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.823278    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms\") pod \"payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:07.823532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.823419    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.823532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.823448    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5k4p\" (UniqueName: \"kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p\") pod \"payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.823676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.823630    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:07.824365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.824335    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.824662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.824638    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:07.842303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.842270    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5k4p\" (UniqueName: \"kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p\") pod \"payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:07.842944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:07.842919    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms\") pod \"payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:08.012538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.012450    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:08.106880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.106842    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:08.319631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.319554    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"]
Oct 30 21:51:08.322747 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:08.322717    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e0a1bc_3914_4fe0_9488_21cb77673acc.slice/crio-c1c25b93c7cc4571c72dfb225d66200d43ad7931633354855e13387a3d2b536b WatchSource:0}: Error finding container c1c25b93c7cc4571c72dfb225d66200d43ad7931633354855e13387a3d2b536b: Status 404 returned error can't find the container with id c1c25b93c7cc4571c72dfb225d66200d43ad7931633354855e13387a3d2b536b
Oct 30 21:51:08.433557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.433529    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"]
Oct 30 21:51:08.436006 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:08.435975    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0566572_3228_448b_8636_249effd583d4.slice/crio-be42fdc42c21aea798fa8736dff1afe6278217d70c8f57768f50dfff337fa70f WatchSource:0}: Error finding container be42fdc42c21aea798fa8736dff1afe6278217d70c8f57768f50dfff337fa70f: Status 404 returned error can't find the container with id be42fdc42c21aea798fa8736dff1afe6278217d70c8f57768f50dfff337fa70f
Oct 30 21:51:08.756615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.756513    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" event={"ID":"a0566572-3228-448b-8636-249effd583d4","Type":"ContainerStarted","Data":"94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82"}
Oct 30 21:51:08.756615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.756571    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" event={"ID":"a0566572-3228-448b-8636-249effd583d4","Type":"ContainerStarted","Data":"be42fdc42c21aea798fa8736dff1afe6278217d70c8f57768f50dfff337fa70f"}
Oct 30 21:51:08.756843 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.756625    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:08.758556 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.758514    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" event={"ID":"b4e0a1bc-3914-4fe0-9488-21cb77673acc","Type":"ContainerStarted","Data":"abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854"}
Oct 30 21:51:08.758719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.758574    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" event={"ID":"b4e0a1bc-3914-4fe0-9488-21cb77673acc","Type":"ContainerStarted","Data":"c1c25b93c7cc4571c72dfb225d66200d43ad7931633354855e13387a3d2b536b"}
Oct 30 21:51:08.758719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.758634    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:08.794835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.794785    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" podStartSLOduration=1.794772956 podStartE2EDuration="1.794772956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:08.793806355 +0000 UTC m=+2451.716264654" watchObservedRunningTime="2026-10-30 21:51:08.794772956 +0000 UTC m=+2451.717231253"
Oct 30 21:51:08.833105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:08.833047    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" podStartSLOduration=1.833032535 podStartE2EDuration="1.833032535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:08.831337663 +0000 UTC m=+2451.753795966" watchObservedRunningTime="2026-10-30 21:51:08.833032535 +0000 UTC m=+2451.755490832"
Oct 30 21:51:13.546277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:13.546246    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:51:15.571627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:15.570784    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:51:15.690072 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:15.690012    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"]
Oct 30 21:51:15.946226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:15.946134    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"]
Oct 30 21:51:19.771334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:19.771302    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 21:51:19.771776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:19.771739    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 21:51:20.078314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:20.078233    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"]
Oct 30 21:51:20.078591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:20.078434    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy" containerID="cri-o://a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066" gracePeriod=30
Oct 30 21:51:20.705577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:20.701718    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn" podUID="71caf68f-87d4-4f22-be9b-4bb1cda97805" containerName="payload-processing" containerID="cri-o://20fd640078f84062e91374f394e586b16b840f1ff81429c9b4d07e7548fdcf0f" gracePeriod=25
Oct 30 21:51:20.888468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:20.888436    6695 generic.go:356] "Generic (PLEG): container finished" podID="71caf68f-87d4-4f22-be9b-4bb1cda97805" containerID="20fd640078f84062e91374f394e586b16b840f1ff81429c9b4d07e7548fdcf0f" exitCode=0
Oct 30 21:51:20.888898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:20.888517    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn" event={"ID":"71caf68f-87d4-4f22-be9b-4bb1cda97805","Type":"ContainerDied","Data":"20fd640078f84062e91374f394e586b16b840f1ff81429c9b4d07e7548fdcf0f"}
Oct 30 21:51:20.947085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:20.947038    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz" podUID="934f90c1-18ae-4376-a57f-eaf17d039e38" containerName="payload-processing" containerID="cri-o://2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0" gracePeriod=25
Oct 30 21:51:21.088451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.088420    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:51:21.183390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.183352    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d\") pod \"71caf68f-87d4-4f22-be9b-4bb1cda97805\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") "
Oct 30 21:51:21.183623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.183432    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume\") pod \"71caf68f-87d4-4f22-be9b-4bb1cda97805\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") "
Oct 30 21:51:21.183623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.183520    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume\") pod \"71caf68f-87d4-4f22-be9b-4bb1cda97805\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") "
Oct 30 21:51:21.183795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.183679    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials\") pod \"71caf68f-87d4-4f22-be9b-4bb1cda97805\" (UID: \"71caf68f-87d4-4f22-be9b-4bb1cda97805\") "
Oct 30 21:51:21.183878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.183847    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume" pod "71caf68f-87d4-4f22-be9b-4bb1cda97805" (UID: "71caf68f-87d4-4f22-be9b-4bb1cda97805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:21.184161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.184134    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume" pod "71caf68f-87d4-4f22-be9b-4bb1cda97805" (UID: "71caf68f-87d4-4f22-be9b-4bb1cda97805"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:21.184373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.184351    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.184468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.184375    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71caf68f-87d4-4f22-be9b-4bb1cda97805-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.186688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.186661    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials" pod "71caf68f-87d4-4f22-be9b-4bb1cda97805" (UID: "71caf68f-87d4-4f22-be9b-4bb1cda97805"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:21.186826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.186700    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d" pod "71caf68f-87d4-4f22-be9b-4bb1cda97805" (UID: "71caf68f-87d4-4f22-be9b-4bb1cda97805"). InnerVolumeSpecName "kube-api-access-7wz7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:21.206852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.206811    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:51:21.285237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.285198    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume\") pod \"934f90c1-18ae-4376-a57f-eaf17d039e38\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") "
Oct 30 21:51:21.285462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.285337    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials\") pod \"934f90c1-18ae-4376-a57f-eaf17d039e38\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") "
Oct 30 21:51:21.286229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.286200    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv\") pod \"934f90c1-18ae-4376-a57f-eaf17d039e38\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") "
Oct 30 21:51:21.286367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.286339    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume\") pod \"934f90c1-18ae-4376-a57f-eaf17d039e38\" (UID: \"934f90c1-18ae-4376-a57f-eaf17d039e38\") "
Oct 30 21:51:21.286490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.286469    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume" pod "934f90c1-18ae-4376-a57f-eaf17d039e38" (UID: "934f90c1-18ae-4376-a57f-eaf17d039e38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:21.287033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.287002    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume" pod "934f90c1-18ae-4376-a57f-eaf17d039e38" (UID: "934f90c1-18ae-4376-a57f-eaf17d039e38"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:21.287468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.287444    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wz7d\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-kube-api-access-7wz7d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.287671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.287472    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.287671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.287486    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71caf68f-87d4-4f22-be9b-4bb1cda97805-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.287671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.287501    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934f90c1-18ae-4376-a57f-eaf17d039e38-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.289236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.289209    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv" pod "934f90c1-18ae-4376-a57f-eaf17d039e38" (UID: "934f90c1-18ae-4376-a57f-eaf17d039e38"). InnerVolumeSpecName "kube-api-access-c25vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:21.291006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.290979    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials" pod "934f90c1-18ae-4376-a57f-eaf17d039e38" (UID: "934f90c1-18ae-4376-a57f-eaf17d039e38"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:21.388887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.388835    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.388887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.388876    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c25vv\" (UniqueName: \"kubernetes.io/projected/934f90c1-18ae-4376-a57f-eaf17d039e38-kube-api-access-c25vv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:21.898742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.898710    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"
Oct 30 21:51:21.898742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.898734    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn" event={"ID":"71caf68f-87d4-4f22-be9b-4bb1cda97805","Type":"ContainerDied","Data":"f3baa366c0048a507946ea875aa987a5299a1c7711084c2912d09215b3e89bf6"}
Oct 30 21:51:21.899272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.898794    6695 scope.go:122] "RemoveContainer" containerID="20fd640078f84062e91374f394e586b16b840f1ff81429c9b4d07e7548fdcf0f"
Oct 30 21:51:21.900670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.900643    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz" event={"ID":"934f90c1-18ae-4376-a57f-eaf17d039e38","Type":"ContainerDied","Data":"2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0"}
Oct 30 21:51:21.900790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.900653    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"
Oct 30 21:51:21.905366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.905338    6695 generic.go:356] "Generic (PLEG): container finished" podID="934f90c1-18ae-4376-a57f-eaf17d039e38" containerID="2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0" exitCode=0
Oct 30 21:51:21.905499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.905408    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz" event={"ID":"934f90c1-18ae-4376-a57f-eaf17d039e38","Type":"ContainerDied","Data":"a24eb355c19b77e528f38a0a06f011165459e03e3e877cee2d119c787cce64b9"}
Oct 30 21:51:21.911445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.911361    6695 scope.go:122] "RemoveContainer" containerID="2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0"
Oct 30 21:51:21.922554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.922529    6695 scope.go:122] "RemoveContainer" containerID="2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0"
Oct 30 21:51:21.922887 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:51:21.922863    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0\": container with ID starting with 2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0 not found: ID does not exist" containerID="2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0"
Oct 30 21:51:21.922943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.922896    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0"} err="failed to get container status \"2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0\": rpc error: code = NotFound desc = could not find container \"2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0\": container with ID starting with 2e1454094e3ec2e41659252106177ade14537776c8b916a9f4cac137d02ef7f0 not found: ID does not exist"
Oct 30 21:51:21.949904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.949871    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"]
Oct 30 21:51:21.969473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.969428    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57kqsmn"]
Oct 30 21:51:21.987384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.987282    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"]
Oct 30 21:51:21.991195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:21.991164    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6qtvwz"]
Oct 30 21:51:22.249195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:22.249115    6695 scope.go:122] "RemoveContainer" containerID="82a8deeb6182cd5d1c5447686d646f07a08aeb81124d76c1adb5649737649216"
Oct 30 21:51:23.216423 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:23.216381    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71caf68f-87d4-4f22-be9b-4bb1cda97805" path="/var/lib/kubelet/pods/71caf68f-87d4-4f22-be9b-4bb1cda97805/volumes"
Oct 30 21:51:23.216991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:23.216971    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934f90c1-18ae-4376-a57f-eaf17d039e38" path="/var/lib/kubelet/pods/934f90c1-18ae-4376-a57f-eaf17d039e38/volumes"
Oct 30 21:51:25.369529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.369500    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:51:25.468526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468483    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.468762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468620    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.468762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468663    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.468762 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468725    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.468976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468898    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.468976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468945    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.468976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:25.469149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.468974    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.469149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469045    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.469149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469122    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy\") pod \"b05f24da-31fd-4428-82f1-42fbe7f7615b\" (UID: \"b05f24da-31fd-4428-82f1-42fbe7f7615b\") "
Oct 30 21:51:25.469323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469286    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:25.469893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469586    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:25.469893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469505    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:25.469893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469843    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:25.470161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469916    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.470161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469937    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.470161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.469952    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.471394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.471356    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:25.471678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.471650    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "kube-api-access-d2zgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:25.471793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.471652    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:51:25.472862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.472826    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy" pod "b05f24da-31fd-4428-82f1-42fbe7f7615b" (UID: "b05f24da-31fd-4428-82f1-42fbe7f7615b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:25.571455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.571416    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.571455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.571452    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2zgp\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-kube-api-access-d2zgp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.571455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.571462    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05f24da-31fd-4428-82f1-42fbe7f7615b-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.571783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.571472    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.571783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.571482    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05f24da-31fd-4428-82f1-42fbe7f7615b-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.571783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.571496    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05f24da-31fd-4428-82f1-42fbe7f7615b-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:51:25.940696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.940653    6695 generic.go:356] "Generic (PLEG): container finished" podID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerID="a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066" exitCode=0
Oct 30 21:51:25.940894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.940706    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"
Oct 30 21:51:25.940894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.940741    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" event={"ID":"b05f24da-31fd-4428-82f1-42fbe7f7615b","Type":"ContainerDied","Data":"a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066"}
Oct 30 21:51:25.940894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.940812    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx" event={"ID":"b05f24da-31fd-4428-82f1-42fbe7f7615b","Type":"ContainerDied","Data":"516a4c4c58aab084f79e76096dc0e306214ab02ed45f3f641e348f6cfdc92466"}
Oct 30 21:51:25.940894 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.940849    6695 scope.go:122] "RemoveContainer" containerID="a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066"
Oct 30 21:51:25.956647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.956619    6695 scope.go:122] "RemoveContainer" containerID="a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066"
Oct 30 21:51:25.956963 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:51:25.956944    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066\": container with ID starting with a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066 not found: ID does not exist" containerID="a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066"
Oct 30 21:51:25.957051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:25.956972    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066"} err="failed to get container status \"a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066\": rpc error: code = NotFound desc = could not find container \"a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066\": container with ID starting with a93a84ced08e14b7d0f8c202d345183581f4cb0e3a93f34212d28ca7e36a0066 not found: ID does not exist"
Oct 30 21:51:26.005966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:26.005933    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"]
Oct 30 21:51:26.017279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:26.017236    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-b7jlx"]
Oct 30 21:51:27.225355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:27.225312    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" path="/var/lib/kubelet/pods/b05f24da-31fd-4428-82f1-42fbe7f7615b/volumes"
Oct 30 21:51:31.031896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:31.031861    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"]
Oct 30 21:51:31.042886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:31.042733    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"]
Oct 30 21:51:31.046541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:31.046517    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386fb9d7-r672p"]
Oct 30 21:51:31.050074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:31.050047    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c960f59-q7b29"]
Oct 30 21:51:31.075852 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:51:31.075810    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:51:31.216741 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:31.216710    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088b97ad-b094-4cc3-bd7d-04ae74d25c23" path="/var/lib/kubelet/pods/088b97ad-b094-4cc3-bd7d-04ae74d25c23/volumes"
Oct 30 21:51:31.217270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:31.217255    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fb5bda-416c-4d9f-87ff-b8b478b5672b" path="/var/lib/kubelet/pods/c6fb5bda-416c-4d9f-87ff-b8b478b5672b/volumes"
Oct 30 21:51:41.028668 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:41.028633    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"]
Oct 30 21:51:41.039615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:41.039574    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306c3e8e-kpdv2"]
Oct 30 21:51:41.215814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:41.215780    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14aa15de-e066-4c2c-af37-f2ae253271f7" path="/var/lib/kubelet/pods/14aa15de-e066-4c2c-af37-f2ae253271f7/volumes"
Oct 30 21:51:42.032013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:42.031966    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"]
Oct 30 21:51:42.042702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:42.042670    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9dcde3b-47ztd"]
Oct 30 21:51:43.216652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:43.216596    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1c8bb7-f75a-4a6f-bb17-602297849595" path="/var/lib/kubelet/pods/5b1c8bb7-f75a-4a6f-bb17-602297849595/volumes"
Oct 30 21:51:49.144042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.143999    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"]
Oct 30 21:51:49.144881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.144856    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71caf68f-87d4-4f22-be9b-4bb1cda97805" containerName="payload-processing"
Oct 30 21:51:49.144881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.144875    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71caf68f-87d4-4f22-be9b-4bb1cda97805" containerName="payload-processing"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.144906    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.144912    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.144926    6695 cpu_manager.go:414] "containerMap: removing container" podUID="934f90c1-18ae-4376-a57f-eaf17d039e38" containerName="payload-processing"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.144931    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934f90c1-18ae-4376-a57f-eaf17d039e38" containerName="payload-processing"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.145036    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71caf68f-87d4-4f22-be9b-4bb1cda97805" containerName="payload-processing"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.145047    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05f24da-31fd-4428-82f1-42fbe7f7615b" containerName="istio-proxy"
Oct 30 21:51:49.145080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.145056    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="934f90c1-18ae-4376-a57f-eaf17d039e38" containerName="payload-processing"
Oct 30 21:51:49.148578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.148560    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.160435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.160392    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"]
Oct 30 21:51:49.162639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.162596    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"payload-processing-external-model-e2e--4472330f-dockercfg-ljrz7\""
Oct 30 21:51:49.162639 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.162628    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"agc-cred-3a8db1af3bcd\""
Oct 30 21:51:49.163081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.163060    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"routing-overlay\""
Oct 30 21:51:49.163236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.163068    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"openshift-service-ca.crt\""
Oct 30 21:51:49.163313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.163127    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"kube-root-ca.crt\""
Oct 30 21:51:49.163313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.163207    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3d5d13\""
Oct 30 21:51:49.245581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.245540    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.245784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.245623    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghlv\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.245784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.245711    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.245784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.245776    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.370630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.369802    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.370630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.369903    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghlv\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.370630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.369943    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.370630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.370020    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.394864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.372004    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.394864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.381409    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.394864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.391675    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.399496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.399461    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghlv\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.471940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.471899    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:49.746840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.746763    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"]
Oct 30 21:51:49.752801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.752775    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.767965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.767940    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"agc-cred-32e88df7aaa5\""
Oct 30 21:51:49.772248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.772218    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"]
Oct 30 21:51:49.801447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.801416    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"]
Oct 30 21:51:49.804983 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:49.804948    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8a993e_ecc9_4cd4_bd93_b96938e8b38d.slice/crio-000c1e0d4a0a5b29894207548aa5b7b4c58bc820ef7f04ba232caabe3a43994e WatchSource:0}: Error finding container 000c1e0d4a0a5b29894207548aa5b7b4c58bc820ef7f04ba232caabe3a43994e: Status 404 returned error can't find the container with id 000c1e0d4a0a5b29894207548aa5b7b4c58bc820ef7f04ba232caabe3a43994e
Oct 30 21:51:49.880246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.880213    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.880407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.880337    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zstp\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.880407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.880368    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.880535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.880412    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.981953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.981909    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zstp\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.982153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.982033    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.982153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.982081    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.982312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.982301    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.982769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.982745    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.983071 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.983045    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:49.985154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:49.985131    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:50.001329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.001268    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zstp\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:50.068301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.068261    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:50.182475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.182437    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk" event={"ID":"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d","Type":"ContainerStarted","Data":"b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d"}
Oct 30 21:51:50.183034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.182496    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk" event={"ID":"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d","Type":"ContainerStarted","Data":"000c1e0d4a0a5b29894207548aa5b7b4c58bc820ef7f04ba232caabe3a43994e"}
Oct 30 21:51:50.183034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.182553    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:51:50.220770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.220720    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk" podStartSLOduration=1.220706434 podStartE2EDuration="1.220706434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:50.220322768 +0000 UTC m=+2493.142781070" watchObservedRunningTime="2026-10-30 21:51:50.220706434 +0000 UTC m=+2493.143164754"
Oct 30 21:51:50.383910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:50.383880    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"]
Oct 30 21:51:50.389063 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:50.389026    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c16d6f7_7fc5_4c1c_a824_07ba261aeda9.slice/crio-798cb8da979fea8b19a26c04e44a556bd4a6fb6b78115f9c37af3267278753b4 WatchSource:0}: Error finding container 798cb8da979fea8b19a26c04e44a556bd4a6fb6b78115f9c37af3267278753b4: Status 404 returned error can't find the container with id 798cb8da979fea8b19a26c04e44a556bd4a6fb6b78115f9c37af3267278753b4
Oct 30 21:51:51.241396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:51.241361    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:51:51.241876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:51.241424    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7" event={"ID":"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9","Type":"ContainerStarted","Data":"36797815398d50890e48a3e98540a6d0f88a4a7a71b636f9d71819fbf3c983b1"}
Oct 30 21:51:51.241876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:51.241452    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7" event={"ID":"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9","Type":"ContainerStarted","Data":"798cb8da979fea8b19a26c04e44a556bd4a6fb6b78115f9c37af3267278753b4"}
Oct 30 21:51:51.264194 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:51.264144    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7" podStartSLOduration=2.264131197 podStartE2EDuration="2.264131197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:51.262927104 +0000 UTC m=+2494.185385403" watchObservedRunningTime="2026-10-30 21:51:51.264131197 +0000 UTC m=+2494.186589495"
Oct 30 21:51:52.066052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:52.066005    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"]
Oct 30 21:51:52.077320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:52.077285    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8a7fc2e-cvxdg"]
Oct 30 21:51:53.217505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:53.217460    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e045e5-0553-4646-bb6f-de1fe94f8d31" path="/var/lib/kubelet/pods/61e045e5-0553-4646-bb6f-de1fe94f8d31/volumes"
Oct 30 21:51:54.108939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.108899    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"]
Oct 30 21:51:54.115201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.115173    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.147713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.147680    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"]
Oct 30 21:51:54.213443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.213404    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgrw\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.213708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.213477    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.213708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.213591    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.213708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.213697    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.315245 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.315208    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.315771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.315272    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.315771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.315331    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.315771 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.315575    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgrw\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.316235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.316210    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.316341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.316303    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.318210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.318188    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.335769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.335738    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgrw\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.431672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.430871    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:54.784616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:54.784572    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"]
Oct 30 21:51:54.786899 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:51:54.786855    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc579486c_fee5_48d9_b0b1_5a6608cfe0fb.slice/crio-114a90fde5e1541ab5ccb4558327c60a3f0cb5bb75838906151e8b1eab5461ec WatchSource:0}: Error finding container 114a90fde5e1541ab5ccb4558327c60a3f0cb5bb75838906151e8b1eab5461ec: Status 404 returned error can't find the container with id 114a90fde5e1541ab5ccb4558327c60a3f0cb5bb75838906151e8b1eab5461ec
Oct 30 21:51:55.258839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:55.258740    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll" event={"ID":"c579486c-fee5-48d9-b0b1-5a6608cfe0fb","Type":"ContainerStarted","Data":"31989ba249e90a5382c5015727eead8f99254da18a03eccf60e3e1a0b07a9056"}
Oct 30 21:51:55.258839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:55.258808    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll" event={"ID":"c579486c-fee5-48d9-b0b1-5a6608cfe0fb","Type":"ContainerStarted","Data":"114a90fde5e1541ab5ccb4558327c60a3f0cb5bb75838906151e8b1eab5461ec"}
Oct 30 21:51:55.258839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:55.258837    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:51:55.300508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:55.300451    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll" podStartSLOduration=1.300436009 podStartE2EDuration="1.300436009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:55.296130818 +0000 UTC m=+2498.218589116" watchObservedRunningTime="2026-10-30 21:51:55.300436009 +0000 UTC m=+2498.222894303"
Oct 30 21:51:56.222003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:56.221970    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"]
Oct 30 21:51:56.225737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:51:56.225712    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:52:01.047783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.047749    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"]
Oct 30 21:52:01.052963 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.052939    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.078079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.078046    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"]
Oct 30 21:52:01.081726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.081616    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:52:01.081726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.081672    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:52:01.081726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.081691    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:01.081726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.081718    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:52:01.083683 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:01.083652    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:52:01.140980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.140927    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.141170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.141138    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.141301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.141281    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6w8\" (UniqueName: \"kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.223047 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.223006    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk" podUID="3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" containerName="payload-processing" containerID="cri-o://b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d" gracePeriod=25
Oct 30 21:52:01.243411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.243369    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.243575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.243534    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.243725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.243706    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6w8\" (UniqueName: \"kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.244129 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.244107    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.245814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.245793    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.270192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.270161    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6w8\" (UniqueName: \"kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8\") pod \"maas-api-revoke-sub-85a9eb00-xsjgq\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.371084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.371050    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:01.501581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.501542    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:52:01.650831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.650716    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials\") pod \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") "
Oct 30 21:52:01.651131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.651098    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume\") pod \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") "
Oct 30 21:52:01.651822 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.651694    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume" pod "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" (UID: "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:01.651969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.651945    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv\") pod \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") "
Oct 30 21:52:01.652037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.652015    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume\") pod \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\" (UID: \"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d\") "
Oct 30 21:52:01.652696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.652670    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume" pod "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" (UID: "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:01.652854 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.652831    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:01.652939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.652858    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:01.654021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.653996    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials" pod "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" (UID: "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:01.659141 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.659115    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv" pod "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" (UID: "3c8a993e-ecc9-4cd4-bd93-b96938e8b38d"). InnerVolumeSpecName "kube-api-access-9ghlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:01.718806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.718773    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"]
Oct 30 21:52:01.722352 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:01.722324    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1259d0e_ff8e_46ae_ab11_b986ff33d51e.slice/crio-98ab8dddc83778631e610122910b4f57224aa441af3e73cb3a903ddefbe5fd24 WatchSource:0}: Error finding container 98ab8dddc83778631e610122910b4f57224aa441af3e73cb3a903ddefbe5fd24: Status 404 returned error can't find the container with id 98ab8dddc83778631e610122910b4f57224aa441af3e73cb3a903ddefbe5fd24
Oct 30 21:52:01.754562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.754530    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghlv\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-kube-api-access-9ghlv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:01.754562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:01.754561    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:02.062673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.062622    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"]
Oct 30 21:52:02.074535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.074497    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a21b8900-42mm4"]
Oct 30 21:52:02.236504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.236426    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:52:02.308161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.308122    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"]
Oct 30 21:52:02.330172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.330137    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq" event={"ID":"d1259d0e-ff8e-46ae-ab11-b986ff33d51e","Type":"ContainerStarted","Data":"280ac4606c32f01311c59115753d7a450acd1c3b1a9d02e24ac1600e7411f538"}
Oct 30 21:52:02.330485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.330439    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq" event={"ID":"d1259d0e-ff8e-46ae-ab11-b986ff33d51e","Type":"ContainerStarted","Data":"98ab8dddc83778631e610122910b4f57224aa441af3e73cb3a903ddefbe5fd24"}
Oct 30 21:52:02.331802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.331749    6695 generic.go:356] "Generic (PLEG): container finished" podID="3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" containerID="b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d" exitCode=0
Oct 30 21:52:02.332180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.332160    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk" event={"ID":"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d","Type":"ContainerDied","Data":"b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d"}
Oct 30 21:52:02.332411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.332376    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"
Oct 30 21:52:02.332563 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.332543    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk" event={"ID":"3c8a993e-ecc9-4cd4-bd93-b96938e8b38d","Type":"ContainerDied","Data":"000c1e0d4a0a5b29894207548aa5b7b4c58bc820ef7f04ba232caabe3a43994e"}
Oct 30 21:52:02.332703 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.332682    6695 scope.go:122] "RemoveContainer" containerID="b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d"
Oct 30 21:52:02.347590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.347555    6695 scope.go:122] "RemoveContainer" containerID="b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d"
Oct 30 21:52:02.348024 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:02.347995    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d\": container with ID starting with b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d not found: ID does not exist" containerID="b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d"
Oct 30 21:52:02.348118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.348034    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d"} err="failed to get container status \"b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d\": rpc error: code = NotFound desc = could not find container \"b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d\": container with ID starting with b50b85ad61fb45d281e119183980dc059c174b8167d835bc7e91a43a00a4b71d not found: ID does not exist"
Oct 30 21:52:02.385059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.385026    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"]
Oct 30 21:52:02.388885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.388854    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6476sqnrk"]
Oct 30 21:52:02.428912 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:02.428858    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq" podStartSLOduration=1.428834635 podStartE2EDuration="1.428834635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:02.425108484 +0000 UTC m=+2505.347566784" watchObservedRunningTime="2026-10-30 21:52:02.428834635 +0000 UTC m=+2505.351292933"
Oct 30 21:52:03.218434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:03.218395    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" path="/var/lib/kubelet/pods/3c8a993e-ecc9-4cd4-bd93-b96938e8b38d/volumes"
Oct 30 21:52:03.219027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:03.219008    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dff4467-a1c4-4d82-8554-f0fa23075681" path="/var/lib/kubelet/pods/3dff4467-a1c4-4d82-8554-f0fa23075681/volumes"
Oct 30 21:52:03.358787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:03.358749    6695 generic.go:356] "Generic (PLEG): container finished" podID="d1259d0e-ff8e-46ae-ab11-b986ff33d51e" containerID="280ac4606c32f01311c59115753d7a450acd1c3b1a9d02e24ac1600e7411f538" exitCode=0
Oct 30 21:52:03.358966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:03.358797    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq" event={"ID":"d1259d0e-ff8e-46ae-ab11-b986ff33d51e","Type":"ContainerDied","Data":"280ac4606c32f01311c59115753d7a450acd1c3b1a9d02e24ac1600e7411f538"}
Oct 30 21:52:04.550979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.550941    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:04.697694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.697573    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8\" (UniqueName: \"kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8\") pod \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") "
Oct 30 21:52:04.697879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.697701    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca\") pod \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") "
Oct 30 21:52:04.697879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.697828    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca\") pod \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\" (UID: \"d1259d0e-ff8e-46ae-ab11-b986ff33d51e\") "
Oct 30 21:52:04.698078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.698052    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca" pod "d1259d0e-ff8e-46ae-ab11-b986ff33d51e" (UID: "d1259d0e-ff8e-46ae-ab11-b986ff33d51e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:04.698508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.698489    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:04.699774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.699751    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8" pod "d1259d0e-ff8e-46ae-ab11-b986ff33d51e" (UID: "d1259d0e-ff8e-46ae-ab11-b986ff33d51e"). InnerVolumeSpecName "kube-api-access-6c6w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:04.699952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.699936    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca" pod "d1259d0e-ff8e-46ae-ab11-b986ff33d51e" (UID: "d1259d0e-ff8e-46ae-ab11-b986ff33d51e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:04.800077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.800037    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6w8\" (UniqueName: \"kubernetes.io/projected/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-kube-api-access-6c6w8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:04.800077 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:04.800073    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1259d0e-ff8e-46ae-ab11-b986ff33d51e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:05.377261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:05.377229    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"
Oct 30 21:52:05.377470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:05.377256    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq" event={"ID":"d1259d0e-ff8e-46ae-ab11-b986ff33d51e","Type":"ContainerDied","Data":"98ab8dddc83778631e610122910b4f57224aa441af3e73cb3a903ddefbe5fd24"}
Oct 30 21:52:05.377470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:05.377318    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ab8dddc83778631e610122910b4f57224aa441af3e73cb3a903ddefbe5fd24"
Oct 30 21:52:06.193352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.193312    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"]
Oct 30 21:52:06.196435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.196405    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:52:06.244062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.244029    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"]
Oct 30 21:52:06.244891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.244873    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" containerName="payload-processing"
Oct 30 21:52:06.244891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.244891    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" containerName="payload-processing"
Oct 30 21:52:06.245038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.244954    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d1259d0e-ff8e-46ae-ab11-b986ff33d51e" containerName="revoke-keys"
Oct 30 21:52:06.245038 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.244960    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1259d0e-ff8e-46ae-ab11-b986ff33d51e" containerName="revoke-keys"
Oct 30 21:52:06.245106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.245070    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1259d0e-ff8e-46ae-ab11-b986ff33d51e" containerName="revoke-keys"
Oct 30 21:52:06.245106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.245079    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8a993e-ecc9-4cd4-bd93-b96938e8b38d" containerName="payload-processing"
Oct 30 21:52:06.249269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.249249    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.273977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.273949    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"]
Oct 30 21:52:06.327210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.327168    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.327447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.327334    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.327540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.327456    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flls8\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.327683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.327660    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.429076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.429037    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.429279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.429123    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.429279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.429168    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flls8\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.429279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.429249    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.429892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.429865    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.430166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.430137    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.431810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.431783    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.449500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.449430    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flls8\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8\") pod \"payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:06.568505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:06.568413    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:07.037154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.036971    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"]
Oct 30 21:52:07.040317 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:07.040284    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55189435_f54b_4dc6_bb85_11878561a1b9.slice/crio-07ff7c815a1acf93f1a25e6d6d98cb56641cb01e721dd1e993cbeefa9acbf3bd WatchSource:0}: Error finding container 07ff7c815a1acf93f1a25e6d6d98cb56641cb01e721dd1e993cbeefa9acbf3bd: Status 404 returned error can't find the container with id 07ff7c815a1acf93f1a25e6d6d98cb56641cb01e721dd1e993cbeefa9acbf3bd
Oct 30 21:52:07.320684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.319963    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx" podUID="2fc422d8-3ab1-463e-b75a-39216939ab8a" containerName="payload-processing" containerID="cri-o://57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe" gracePeriod=25
Oct 30 21:52:07.395387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.395347    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v" event={"ID":"55189435-f54b-4dc6-bb85-11878561a1b9","Type":"ContainerStarted","Data":"ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac"}
Oct 30 21:52:07.395560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.395415    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v" event={"ID":"55189435-f54b-4dc6-bb85-11878561a1b9","Type":"ContainerStarted","Data":"07ff7c815a1acf93f1a25e6d6d98cb56641cb01e721dd1e993cbeefa9acbf3bd"}
Oct 30 21:52:07.395560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.395443    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:07.436881 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.436806    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v" podStartSLOduration=1.43678585 podStartE2EDuration="1.43678585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:07.434947526 +0000 UTC m=+2510.357405825" watchObservedRunningTime="2026-10-30 21:52:07.43678585 +0000 UTC m=+2510.359244149"
Oct 30 21:52:07.625168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.625143    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:52:07.759588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.759552    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume\") pod \"2fc422d8-3ab1-463e-b75a-39216939ab8a\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") "
Oct 30 21:52:07.759806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.759633    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials\") pod \"2fc422d8-3ab1-463e-b75a-39216939ab8a\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") "
Oct 30 21:52:07.759806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.759798    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n\") pod \"2fc422d8-3ab1-463e-b75a-39216939ab8a\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") "
Oct 30 21:52:07.759917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.759816    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume\") pod \"2fc422d8-3ab1-463e-b75a-39216939ab8a\" (UID: \"2fc422d8-3ab1-463e-b75a-39216939ab8a\") "
Oct 30 21:52:07.760062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.760029    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume" pod "2fc422d8-3ab1-463e-b75a-39216939ab8a" (UID: "2fc422d8-3ab1-463e-b75a-39216939ab8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:07.760256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.760231    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume" pod "2fc422d8-3ab1-463e-b75a-39216939ab8a" (UID: "2fc422d8-3ab1-463e-b75a-39216939ab8a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:07.760409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.760392    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:07.760466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.760414    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fc422d8-3ab1-463e-b75a-39216939ab8a-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:07.762087 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.762059    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n" pod "2fc422d8-3ab1-463e-b75a-39216939ab8a" (UID: "2fc422d8-3ab1-463e-b75a-39216939ab8a"). InnerVolumeSpecName "kube-api-access-v574n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:07.762205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.762089    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials" pod "2fc422d8-3ab1-463e-b75a-39216939ab8a" (UID: "2fc422d8-3ab1-463e-b75a-39216939ab8a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:07.862275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.862195    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v574n\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-kube-api-access-v574n\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:07.862275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:07.862225    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fc422d8-3ab1-463e-b75a-39216939ab8a-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:08.404488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.404453    6695 generic.go:356] "Generic (PLEG): container finished" podID="2fc422d8-3ab1-463e-b75a-39216939ab8a" containerID="57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe" exitCode=0
Oct 30 21:52:08.404998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.404493    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"
Oct 30 21:52:08.404998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.404536    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx" event={"ID":"2fc422d8-3ab1-463e-b75a-39216939ab8a","Type":"ContainerDied","Data":"57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe"}
Oct 30 21:52:08.404998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.404614    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx" event={"ID":"2fc422d8-3ab1-463e-b75a-39216939ab8a","Type":"ContainerDied","Data":"7edbe624862466915e69046eca2d987ccfe40fa2a27529e44ddac23aa8c0e227"}
Oct 30 21:52:08.404998 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.404651    6695 scope.go:122] "RemoveContainer" containerID="57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe"
Oct 30 21:52:08.420428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.420394    6695 scope.go:122] "RemoveContainer" containerID="57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe"
Oct 30 21:52:08.420838 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:08.420816    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe\": container with ID starting with 57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe not found: ID does not exist" containerID="57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe"
Oct 30 21:52:08.420948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.420845    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe"} err="failed to get container status \"57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe\": rpc error: code = NotFound desc = could not find container \"57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe\": container with ID starting with 57b78c4537e1ae9984cc956a22a1f15c859c1bcfa65e56da0a22a787f0be3cfe not found: ID does not exist"
Oct 30 21:52:08.467702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.467668    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"]
Oct 30 21:52:08.478656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:08.478617    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669j92xx"]
Oct 30 21:52:09.235638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:09.235589    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc422d8-3ab1-463e-b75a-39216939ab8a" path="/var/lib/kubelet/pods/2fc422d8-3ab1-463e-b75a-39216939ab8a/volumes"
Oct 30 21:52:11.194126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.194077    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll" podUID="c579486c-fee5-48d9-b0b1-5a6608cfe0fb" containerName="payload-processing" containerID="cri-o://31989ba249e90a5382c5015727eead8f99254da18a03eccf60e3e1a0b07a9056" gracePeriod=25
Oct 30 21:52:11.436739 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.436700    6695 generic.go:356] "Generic (PLEG): container finished" podID="c579486c-fee5-48d9-b0b1-5a6608cfe0fb" containerID="31989ba249e90a5382c5015727eead8f99254da18a03eccf60e3e1a0b07a9056" exitCode=0
Oct 30 21:52:11.436917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.436772    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll" event={"ID":"c579486c-fee5-48d9-b0b1-5a6608cfe0fb","Type":"ContainerDied","Data":"31989ba249e90a5382c5015727eead8f99254da18a03eccf60e3e1a0b07a9056"}
Oct 30 21:52:11.477126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.477090    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:52:11.620337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.620303    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials\") pod \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") "
Oct 30 21:52:11.620519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.620379    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw\") pod \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") "
Oct 30 21:52:11.620699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.620672    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume\") pod \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") "
Oct 30 21:52:11.620829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.620739    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume\") pod \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\" (UID: \"c579486c-fee5-48d9-b0b1-5a6608cfe0fb\") "
Oct 30 21:52:11.621069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.621039    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume" pod "c579486c-fee5-48d9-b0b1-5a6608cfe0fb" (UID: "c579486c-fee5-48d9-b0b1-5a6608cfe0fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:11.621174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.621102    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume" pod "c579486c-fee5-48d9-b0b1-5a6608cfe0fb" (UID: "c579486c-fee5-48d9-b0b1-5a6608cfe0fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:11.621485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.621462    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:11.621576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.621494    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:11.622791 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.622763    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw" pod "c579486c-fee5-48d9-b0b1-5a6608cfe0fb" (UID: "c579486c-fee5-48d9-b0b1-5a6608cfe0fb"). InnerVolumeSpecName "kube-api-access-nxgrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:11.622882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.622783    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials" pod "c579486c-fee5-48d9-b0b1-5a6608cfe0fb" (UID: "c579486c-fee5-48d9-b0b1-5a6608cfe0fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:11.723413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.723323    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:11.723413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:11.723356    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxgrw\" (UniqueName: \"kubernetes.io/projected/c579486c-fee5-48d9-b0b1-5a6608cfe0fb-kube-api-access-nxgrw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:12.410800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:12.410765    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"]
Oct 30 21:52:12.457057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:12.457019    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll" event={"ID":"c579486c-fee5-48d9-b0b1-5a6608cfe0fb","Type":"ContainerDied","Data":"114a90fde5e1541ab5ccb4558327c60a3f0cb5bb75838906151e8b1eab5461ec"}
Oct 30 21:52:12.457326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:12.457312    6695 scope.go:122] "RemoveContainer" containerID="31989ba249e90a5382c5015727eead8f99254da18a03eccf60e3e1a0b07a9056"
Oct 30 21:52:12.457512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:12.457080    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"
Oct 30 21:52:12.516805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:12.516773    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"]
Oct 30 21:52:12.521076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:12.521047    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6c57tk7ll"]
Oct 30 21:52:13.218697 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:13.218659    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c579486c-fee5-48d9-b0b1-5a6608cfe0fb" path="/var/lib/kubelet/pods/c579486c-fee5-48d9-b0b1-5a6608cfe0fb/volumes"
Oct 30 21:52:15.201947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.201906    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"]
Oct 30 21:52:15.202765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.202749    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2fc422d8-3ab1-463e-b75a-39216939ab8a" containerName="payload-processing"
Oct 30 21:52:15.202765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.202765    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc422d8-3ab1-463e-b75a-39216939ab8a" containerName="payload-processing"
Oct 30 21:52:15.202867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.202813    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c579486c-fee5-48d9-b0b1-5a6608cfe0fb" containerName="payload-processing"
Oct 30 21:52:15.202867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.202818    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c579486c-fee5-48d9-b0b1-5a6608cfe0fb" containerName="payload-processing"
Oct 30 21:52:15.202940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.202935    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c579486c-fee5-48d9-b0b1-5a6608cfe0fb" containerName="payload-processing"
Oct 30 21:52:15.202983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.202947    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc422d8-3ab1-463e-b75a-39216939ab8a" containerName="payload-processing"
Oct 30 21:52:15.206733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.206701    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.226073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.226029    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:15.226502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.226404    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:52:15.228811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.228780    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"]
Oct 30 21:52:15.235000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.234969    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:52:15.235000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.234985    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:52:15.271763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.271718    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.272131 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.272107    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.272299 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.272273    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w5nv\" (UniqueName: \"kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.374033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.373995    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.374195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.374066    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.374195 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.374138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w5nv\" (UniqueName: \"kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.374823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.374786    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.376434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.376405    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.404479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.404447    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w5nv\" (UniqueName: \"kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv\") pod \"maas-api-revoke-sub-3fb79667-b5wgj\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.522024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.521993    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:15.839509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:15.839346    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"]
Oct 30 21:52:15.843915 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:15.843880    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d534e28_ec0d_42fa_976e_af7e63d20caf.slice/crio-31a1f7510d49ee9ac7896bb1b2ffdb8c2ea7937822ac2789338c7048f6ec0348 WatchSource:0}: Error finding container 31a1f7510d49ee9ac7896bb1b2ffdb8c2ea7937822ac2789338c7048f6ec0348: Status 404 returned error can't find the container with id 31a1f7510d49ee9ac7896bb1b2ffdb8c2ea7937822ac2789338c7048f6ec0348
Oct 30 21:52:16.493323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:16.493217    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj" event={"ID":"5d534e28-ec0d-42fa-976e-af7e63d20caf","Type":"ContainerStarted","Data":"9e50362acdac4a623f4eab31ae609392d396a8f2dd5babce7671a25a0a531868"}
Oct 30 21:52:16.493323 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:16.493284    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj" event={"ID":"5d534e28-ec0d-42fa-976e-af7e63d20caf","Type":"ContainerStarted","Data":"31a1f7510d49ee9ac7896bb1b2ffdb8c2ea7937822ac2789338c7048f6ec0348"}
Oct 30 21:52:16.543126 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:16.543066    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj" podStartSLOduration=1.5430482620000001 podStartE2EDuration="1.543048262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:16.537305951 +0000 UTC m=+2519.459764261" watchObservedRunningTime="2026-10-30 21:52:16.543048262 +0000 UTC m=+2519.465506559"
Oct 30 21:52:17.411798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.411744    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg" podUID="81ca39d4-0300-491c-8625-aa0642eec4e1" containerName="payload-processing" containerID="cri-o://e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2" gracePeriod=25
Oct 30 21:52:17.502907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.502862    6695 generic.go:356] "Generic (PLEG): container finished" podID="5d534e28-ec0d-42fa-976e-af7e63d20caf" containerID="9e50362acdac4a623f4eab31ae609392d396a8f2dd5babce7671a25a0a531868" exitCode=0
Oct 30 21:52:17.503347 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.502950    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj" event={"ID":"5d534e28-ec0d-42fa-976e-af7e63d20caf","Type":"ContainerDied","Data":"9e50362acdac4a623f4eab31ae609392d396a8f2dd5babce7671a25a0a531868"}
Oct 30 21:52:17.721553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.721523    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:52:17.813302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.813266    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume\") pod \"81ca39d4-0300-491c-8625-aa0642eec4e1\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") "
Oct 30 21:52:17.813481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.813322    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg\") pod \"81ca39d4-0300-491c-8625-aa0642eec4e1\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") "
Oct 30 21:52:17.813481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.813367    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume\") pod \"81ca39d4-0300-491c-8625-aa0642eec4e1\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") "
Oct 30 21:52:17.813481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.813466    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials\") pod \"81ca39d4-0300-491c-8625-aa0642eec4e1\" (UID: \"81ca39d4-0300-491c-8625-aa0642eec4e1\") "
Oct 30 21:52:17.813832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.813786    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume" pod "81ca39d4-0300-491c-8625-aa0642eec4e1" (UID: "81ca39d4-0300-491c-8625-aa0642eec4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:17.813964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.813837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume" pod "81ca39d4-0300-491c-8625-aa0642eec4e1" (UID: "81ca39d4-0300-491c-8625-aa0642eec4e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:17.814219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.814200    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:17.814219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.814221    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81ca39d4-0300-491c-8625-aa0642eec4e1-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:17.815736 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.815714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg" pod "81ca39d4-0300-491c-8625-aa0642eec4e1" (UID: "81ca39d4-0300-491c-8625-aa0642eec4e1"). InnerVolumeSpecName "kube-api-access-8qwlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:17.815839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.815756    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials" pod "81ca39d4-0300-491c-8625-aa0642eec4e1" (UID: "81ca39d4-0300-491c-8625-aa0642eec4e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:17.915921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.915885    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:17.915921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:17.915914    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwlg\" (UniqueName: \"kubernetes.io/projected/81ca39d4-0300-491c-8625-aa0642eec4e1-kube-api-access-8qwlg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:18.408085 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.408054    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:18.507673 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.507635    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"]
Oct 30 21:52:18.514187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.514154    6695 generic.go:356] "Generic (PLEG): container finished" podID="81ca39d4-0300-491c-8625-aa0642eec4e1" containerID="e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2" exitCode=0
Oct 30 21:52:18.514360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.514206    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"
Oct 30 21:52:18.514360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.514239    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg" event={"ID":"81ca39d4-0300-491c-8625-aa0642eec4e1","Type":"ContainerDied","Data":"e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2"}
Oct 30 21:52:18.514360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.514295    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg" event={"ID":"81ca39d4-0300-491c-8625-aa0642eec4e1","Type":"ContainerDied","Data":"ad83bc31f12a5d7646d1fe6a289e8136145d682098e477544eb6688169d8c1b9"}
Oct 30 21:52:18.514360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.514318    6695 scope.go:122] "RemoveContainer" containerID="e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2"
Oct 30 21:52:18.528294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.528267    6695 scope.go:122] "RemoveContainer" containerID="e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2"
Oct 30 21:52:18.528679 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:18.528649    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2\": container with ID starting with e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2 not found: ID does not exist" containerID="e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2"
Oct 30 21:52:18.528775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.528691    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2"} err="failed to get container status \"e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2\": rpc error: code = NotFound desc = could not find container \"e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2\": container with ID starting with e10a3c0aabcda4b6d4ab12f0357442bdb6e40cb1f117684f6ac4a48ff0dbf8b2 not found: ID does not exist"
Oct 30 21:52:18.615546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.615503    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"]
Oct 30 21:52:18.621250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.620723    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9722ac/payload-processing-external-model-e2e-extmod-b-9722ac-d944sdgfg"]
Oct 30 21:52:18.724908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.724882    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:18.826324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.826292    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv\" (UniqueName: \"kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv\") pod \"5d534e28-ec0d-42fa-976e-af7e63d20caf\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") "
Oct 30 21:52:18.826549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.826531    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca\") pod \"5d534e28-ec0d-42fa-976e-af7e63d20caf\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") "
Oct 30 21:52:18.826686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.826665    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca\") pod \"5d534e28-ec0d-42fa-976e-af7e63d20caf\" (UID: \"5d534e28-ec0d-42fa-976e-af7e63d20caf\") "
Oct 30 21:52:18.827025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.826993    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca" pod "5d534e28-ec0d-42fa-976e-af7e63d20caf" (UID: "5d534e28-ec0d-42fa-976e-af7e63d20caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:18.827294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.827273    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:18.828521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.828492    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv" pod "5d534e28-ec0d-42fa-976e-af7e63d20caf" (UID: "5d534e28-ec0d-42fa-976e-af7e63d20caf"). InnerVolumeSpecName "kube-api-access-9w5nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:18.828713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.828695    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca" pod "5d534e28-ec0d-42fa-976e-af7e63d20caf" (UID: "5d534e28-ec0d-42fa-976e-af7e63d20caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:18.928272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.928232    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d534e28-ec0d-42fa-976e-af7e63d20caf-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:18.928272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:18.928267    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w5nv\" (UniqueName: \"kubernetes.io/projected/5d534e28-ec0d-42fa-976e-af7e63d20caf-kube-api-access-9w5nv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:19.216298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:19.216265    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ca39d4-0300-491c-8625-aa0642eec4e1" path="/var/lib/kubelet/pods/81ca39d4-0300-491c-8625-aa0642eec4e1/volumes"
Oct 30 21:52:19.523571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:19.523539    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"
Oct 30 21:52:19.524054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:19.523570    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj" event={"ID":"5d534e28-ec0d-42fa-976e-af7e63d20caf","Type":"ContainerDied","Data":"31a1f7510d49ee9ac7896bb1b2ffdb8c2ea7937822ac2789338c7048f6ec0348"}
Oct 30 21:52:19.524054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:19.523641    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a1f7510d49ee9ac7896bb1b2ffdb8c2ea7937822ac2789338c7048f6ec0348"
Oct 30 21:52:22.299767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:22.299729    6695 scope.go:122] "RemoveContainer" containerID="7634c6a5b71bb4f560065f0a431401982582914e0209292ecccb8f3408108b21"
Oct 30 21:52:22.311537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:22.311513    6695 scope.go:122] "RemoveContainer" containerID="ac9ddc0bb49507308c0a498dc655ab7a506064f58e5734303430bd69d7f1d93e"
Oct 30 21:52:23.508902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.508836    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8" podUID="f291c364-1cea-464b-aad9-9637fdb5d474" containerName="payload-processing" containerID="cri-o://9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513" gracePeriod=25
Oct 30 21:52:23.786955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.786920    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:52:23.896265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896224    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume\") pod \"f291c364-1cea-464b-aad9-9637fdb5d474\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") "
Oct 30 21:52:23.896445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896285    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh\") pod \"f291c364-1cea-464b-aad9-9637fdb5d474\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") "
Oct 30 21:52:23.896445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896342    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume\") pod \"f291c364-1cea-464b-aad9-9637fdb5d474\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") "
Oct 30 21:52:23.896445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896405    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials\") pod \"f291c364-1cea-464b-aad9-9637fdb5d474\" (UID: \"f291c364-1cea-464b-aad9-9637fdb5d474\") "
Oct 30 21:52:23.896754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896684    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume" pod "f291c364-1cea-464b-aad9-9637fdb5d474" (UID: "f291c364-1cea-464b-aad9-9637fdb5d474"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:23.896754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896723    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume" pod "f291c364-1cea-464b-aad9-9637fdb5d474" (UID: "f291c364-1cea-464b-aad9-9637fdb5d474"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:23.896986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896969    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:23.897037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.896989    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f291c364-1cea-464b-aad9-9637fdb5d474-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:23.898519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.898495    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials" pod "f291c364-1cea-464b-aad9-9637fdb5d474" (UID: "f291c364-1cea-464b-aad9-9637fdb5d474"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:23.898623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.898579    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh" pod "f291c364-1cea-464b-aad9-9637fdb5d474" (UID: "f291c364-1cea-464b-aad9-9637fdb5d474"). InnerVolumeSpecName "kube-api-access-zmvmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:23.998369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.998337    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvmh\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-kube-api-access-zmvmh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:23.998369 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:23.998367    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f291c364-1cea-464b-aad9-9637fdb5d474-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:24.564165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.564122    6695 generic.go:356] "Generic (PLEG): container finished" podID="f291c364-1cea-464b-aad9-9637fdb5d474" containerID="9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513" exitCode=0
Oct 30 21:52:24.564695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.564170    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8" event={"ID":"f291c364-1cea-464b-aad9-9637fdb5d474","Type":"ContainerDied","Data":"9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513"}
Oct 30 21:52:24.564695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.564240    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8" event={"ID":"f291c364-1cea-464b-aad9-9637fdb5d474","Type":"ContainerDied","Data":"53e966c05f286f7d6bccbb276a42ce8efde91a746a3a295823eeb8ebb28136c9"}
Oct 30 21:52:24.564695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.564181    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"
Oct 30 21:52:24.564695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.564270    6695 scope.go:122] "RemoveContainer" containerID="9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513"
Oct 30 21:52:24.577300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.577277    6695 scope.go:122] "RemoveContainer" containerID="9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513"
Oct 30 21:52:24.577664 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:24.577642    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513\": container with ID starting with 9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513 not found: ID does not exist" containerID="9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513"
Oct 30 21:52:24.577751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.577674    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513"} err="failed to get container status \"9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513\": rpc error: code = NotFound desc = could not find container \"9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513\": container with ID starting with 9758234ca1c873d60a3e310601db63c246764c9753632be911d6647c09c74513 not found: ID does not exist"
Oct 30 21:52:24.618754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.618723    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"]
Oct 30 21:52:24.630114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.630084    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-65cc4g4v8"]
Oct 30 21:52:24.959931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.959842    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"]
Oct 30 21:52:24.960130 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:24.960102    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy" containerID="cri-o://5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787" gracePeriod=30
Oct 30 21:52:25.216033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.215949    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f291c364-1cea-464b-aad9-9637fdb5d474" path="/var/lib/kubelet/pods/f291c364-1cea-464b-aad9-9637fdb5d474/volumes"
Oct 30 21:52:25.758327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.758290    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"]
Oct 30 21:52:25.759192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759158    6695 cpu_manager.go:414] "containerMap: removing container" podUID="81ca39d4-0300-491c-8625-aa0642eec4e1" containerName="payload-processing"
Oct 30 21:52:25.759192 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759191    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ca39d4-0300-491c-8625-aa0642eec4e1" containerName="payload-processing"
Oct 30 21:52:25.759342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759229    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f291c364-1cea-464b-aad9-9637fdb5d474" containerName="payload-processing"
Oct 30 21:52:25.759342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759236    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f291c364-1cea-464b-aad9-9637fdb5d474" containerName="payload-processing"
Oct 30 21:52:25.759342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759247    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5d534e28-ec0d-42fa-976e-af7e63d20caf" containerName="revoke-keys"
Oct 30 21:52:25.759342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759252    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d534e28-ec0d-42fa-976e-af7e63d20caf" containerName="revoke-keys"
Oct 30 21:52:25.759491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759395    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ca39d4-0300-491c-8625-aa0642eec4e1" containerName="payload-processing"
Oct 30 21:52:25.759491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759409    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f291c364-1cea-464b-aad9-9637fdb5d474" containerName="payload-processing"
Oct 30 21:52:25.759491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.759418    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d534e28-ec0d-42fa-976e-af7e63d20caf" containerName="revoke-keys"
Oct 30 21:52:25.764905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.764883    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.783026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.782994    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"]
Oct 30 21:52:25.789841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.789817    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:52:25.789841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.789828    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:25.789841 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.789813    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:52:25.791839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.791815    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:52:25.834125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.834080    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.834327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.834297    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.834459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.834438    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8hg\" (UniqueName: \"kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.936173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.936132    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8hg\" (UniqueName: \"kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.936399 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.936249    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.936399 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.936363    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.936974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.936953    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.938837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.938808    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:25.956598 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:25.956567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8hg\" (UniqueName: \"kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg\") pod \"maas-api-revoke-sub-f2717da8-vkkxf\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:26.080416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:26.080376    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:26.391689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:26.391664    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"]
Oct 30 21:52:26.394453 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:26.394419    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98d5d9f_e412_44f3_8432_22bd319d351b.slice/crio-ddd6a45649f48e17d304e5a153c1e568bb5e25048a7de3e8355f363d393750b0 WatchSource:0}: Error finding container ddd6a45649f48e17d304e5a153c1e568bb5e25048a7de3e8355f363d393750b0: Status 404 returned error can't find the container with id ddd6a45649f48e17d304e5a153c1e568bb5e25048a7de3e8355f363d393750b0
Oct 30 21:52:26.586215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:26.586180    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf" event={"ID":"b98d5d9f-e412-44f3-8432-22bd319d351b","Type":"ContainerStarted","Data":"ddd6a45649f48e17d304e5a153c1e568bb5e25048a7de3e8355f363d393750b0"}
Oct 30 21:52:27.596250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:27.596153    6695 generic.go:356] "Generic (PLEG): container finished" podID="b98d5d9f-e412-44f3-8432-22bd319d351b" containerID="e0fa2e056f2530d6257625629362760ed7c54370c629e8cb6cfc4cea408f8cd0" exitCode=0
Oct 30 21:52:27.596705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:27.596400    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf" event={"ID":"b98d5d9f-e412-44f3-8432-22bd319d351b","Type":"ContainerDied","Data":"e0fa2e056f2530d6257625629362760ed7c54370c629e8cb6cfc4cea408f8cd0"}
Oct 30 21:52:28.769956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.769921    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:28.908147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.908026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca\") pod \"b98d5d9f-e412-44f3-8432-22bd319d351b\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") "
Oct 30 21:52:28.908333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.908172    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca\") pod \"b98d5d9f-e412-44f3-8432-22bd319d351b\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") "
Oct 30 21:52:28.908333 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.908210    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg\" (UniqueName: \"kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg\") pod \"b98d5d9f-e412-44f3-8432-22bd319d351b\" (UID: \"b98d5d9f-e412-44f3-8432-22bd319d351b\") "
Oct 30 21:52:28.908469 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.908388    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca" pod "b98d5d9f-e412-44f3-8432-22bd319d351b" (UID: "b98d5d9f-e412-44f3-8432-22bd319d351b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:28.908718 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.908697    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:28.910462 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.910433    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca" pod "b98d5d9f-e412-44f3-8432-22bd319d351b" (UID: "b98d5d9f-e412-44f3-8432-22bd319d351b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:28.910580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:28.910548    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg" pod "b98d5d9f-e412-44f3-8432-22bd319d351b" (UID: "b98d5d9f-e412-44f3-8432-22bd319d351b"). InnerVolumeSpecName "kube-api-access-th8hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:29.010108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:29.010068    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98d5d9f-e412-44f3-8432-22bd319d351b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:29.010108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:29.010109    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th8hg\" (UniqueName: \"kubernetes.io/projected/b98d5d9f-e412-44f3-8432-22bd319d351b-kube-api-access-th8hg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:29.614151 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:29.614118    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"
Oct 30 21:52:29.614356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:29.614147    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf" event={"ID":"b98d5d9f-e412-44f3-8432-22bd319d351b","Type":"ContainerDied","Data":"ddd6a45649f48e17d304e5a153c1e568bb5e25048a7de3e8355f363d393750b0"}
Oct 30 21:52:29.614356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:29.614202    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd6a45649f48e17d304e5a153c1e568bb5e25048a7de3e8355f363d393750b0"
Oct 30 21:52:30.241622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.241529    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:52:30.334746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.334708    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.334953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.334771    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.334953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.334828    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.334953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.334860    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.334953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.334896    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.334953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.334936    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.335252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.335116    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.335252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.335144    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.335252 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.335183    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket\") pod \"57bbe453-77e5-454a-88c1-073ee6b6e94c\" (UID: \"57bbe453-77e5-454a-88c1-073ee6b6e94c\") "
Oct 30 21:52:30.336412 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.336379    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:30.337271 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.337246    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:30.338471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.338011    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:30.338471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.338336    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:30.346996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.346951    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:30.349525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.349498    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:30.349721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.349698    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:52:30.350711 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.350684    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:30.361620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.359729    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k" pod "57bbe453-77e5-454a-88c1-073ee6b6e94c" (UID: "57bbe453-77e5-454a-88c1-073ee6b6e94c"). InnerVolumeSpecName "kube-api-access-pv29k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:30.437626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437554    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437597    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bbe453-77e5-454a-88c1-073ee6b6e94c-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437639    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437648    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437659    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437667    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437675    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437683    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bbe453-77e5-454a-88c1-073ee6b6e94c-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.437852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.437717    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv29k\" (UniqueName: \"kubernetes.io/projected/57bbe453-77e5-454a-88c1-073ee6b6e94c-kube-api-access-pv29k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:30.624282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.624241    6695 generic.go:356] "Generic (PLEG): container finished" podID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerID="5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787" exitCode=0
Oct 30 21:52:30.624444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.624301    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"
Oct 30 21:52:30.624444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.624327    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" event={"ID":"57bbe453-77e5-454a-88c1-073ee6b6e94c","Type":"ContainerDied","Data":"5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787"}
Oct 30 21:52:30.624444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.624415    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt" event={"ID":"57bbe453-77e5-454a-88c1-073ee6b6e94c","Type":"ContainerDied","Data":"b523830c067d1a3bc89aecd02e300f6d4862f97393c64b97f10b224dd29ac7ea"}
Oct 30 21:52:30.624444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.624439    6695 scope.go:122] "RemoveContainer" containerID="5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787"
Oct 30 21:52:30.637774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.637746    6695 scope.go:122] "RemoveContainer" containerID="5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787"
Oct 30 21:52:30.638094 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:30.638072    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787\": container with ID starting with 5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787 not found: ID does not exist" containerID="5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787"
Oct 30 21:52:30.638172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.638108    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787"} err="failed to get container status \"5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787\": rpc error: code = NotFound desc = could not find container \"5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787\": container with ID starting with 5e06acc28e9b3aee9c52e035f06bceff3d4f1651e20c00f068c3a9959dfff787 not found: ID does not exist"
Oct 30 21:52:30.700164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.700129    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"]
Oct 30 21:52:30.713630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.706407    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-xpwvt"]
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.724679    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"]
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.726248    6695 cpu_manager.go:414] "containerMap: removing container" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy"
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.726292    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy"
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.726368    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b98d5d9f-e412-44f3-8432-22bd319d351b" containerName="revoke-keys"
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.726377    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98d5d9f-e412-44f3-8432-22bd319d351b" containerName="revoke-keys"
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.726625    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98d5d9f-e412-44f3-8432-22bd319d351b" containerName="revoke-keys"
Oct 30 21:52:30.728626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.726673    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" containerName="istio-proxy"
Oct 30 21:52:30.762885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.760426    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.770647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.768825    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"]
Oct 30 21:52:30.781001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.780966    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-805edc\""
Oct 30 21:52:30.781001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.780983    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"routing-overlay\""
Oct 30 21:52:30.781250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.781086    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"agc-cred-5467e7a748c7\""
Oct 30 21:52:30.781250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.781088    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"openshift-service-ca.crt\""
Oct 30 21:52:30.781519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.781497    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"payload-processing-external-model-e2e--c4b58ec6-dockercfg-hxwmh\""
Oct 30 21:52:30.781941 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.781923    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"kube-root-ca.crt\""
Oct 30 21:52:30.846060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.846005    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.846236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.846116    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.846236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.846152    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.846384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.846362    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4cf\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.965698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.948083    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4cf\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.965698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.948280    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.965698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.948333    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.965698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.948365    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.965698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.949271    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.965698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.950344    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.980625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.978708    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:30.998425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:30.998355    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4cf\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774b999b6\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:31.077193 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:31.077156    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:52:31.087553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.087518    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:31.219204 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.219111    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bbe453-77e5-454a-88c1-073ee6b6e94c" path="/var/lib/kubelet/pods/57bbe453-77e5-454a-88c1-073ee6b6e94c/volumes"
Oct 30 21:52:31.405255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.405021    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"]
Oct 30 21:52:31.408295 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:31.408256    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8bd0b9_7dbc_4bab_83d4_18425e9c3575.slice/crio-dc725af8f53ec9b220b2968acff4aa07f2b1a71719fe72817ba8a284d7d185f7 WatchSource:0}: Error finding container dc725af8f53ec9b220b2968acff4aa07f2b1a71719fe72817ba8a284d7d185f7: Status 404 returned error can't find the container with id dc725af8f53ec9b220b2968acff4aa07f2b1a71719fe72817ba8a284d7d185f7
Oct 30 21:52:31.636206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.636167    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" event={"ID":"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575","Type":"ContainerStarted","Data":"bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe"}
Oct 30 21:52:31.636509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.636223    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" event={"ID":"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575","Type":"ContainerStarted","Data":"dc725af8f53ec9b220b2968acff4aa07f2b1a71719fe72817ba8a284d7d185f7"}
Oct 30 21:52:31.636509 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.636297    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:31.679906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:31.679847    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" podStartSLOduration=1.679834053 podStartE2EDuration="1.679834053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:31.678789758 +0000 UTC m=+2534.601248056" watchObservedRunningTime="2026-10-30 21:52:31.679834053 +0000 UTC m=+2534.602292351"
Oct 30 21:52:31.760891 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:31.760801    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:31.760891 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:31.760832    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:31.761327 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:31.760941    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials podName:3d8bd0b9-7dbc-4bab-83d4-18425e9c3575 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:32.260916696 +0000 UTC m=+2535.183374982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:32.269700 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:32.269660    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:32.269700 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:32.269691    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:32.269937 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:32.269747    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials podName:3d8bd0b9-7dbc-4bab-83d4-18425e9c3575 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:33.269731783 +0000 UTC m=+2536.192190059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:32.512684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.512142    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"]
Oct 30 21:52:32.535719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.527395    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.574629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.542322    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"]
Oct 30 21:52:32.574629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.548624    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"e2e-cfl-3e7cf6-provider-cred\""
Oct 30 21:52:32.684785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.684752    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.684785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.684789    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.685035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.684903    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.685171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.685124    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.786404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.786317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.787471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.787440    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.787759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.787738    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.787921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.787778    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.818786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.788672    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.818786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.789898    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.847681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.837366    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.861624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.859312    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:32.887196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:32.887110    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:33.246679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:33.246582    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"]
Oct 30 21:52:33.249463 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:33.249437    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa238f7_b340_4e88_b88f_29120c68a52a.slice/crio-a3a47ae998da3d5c3c29f9f2b1e3b3f5b72390fa545529b9c9ebdf72f9f70a2f WatchSource:0}: Error finding container a3a47ae998da3d5c3c29f9f2b1e3b3f5b72390fa545529b9c9ebdf72f9f70a2f: Status 404 returned error can't find the container with id a3a47ae998da3d5c3c29f9f2b1e3b3f5b72390fa545529b9c9ebdf72f9f70a2f
Oct 30 21:52:33.300969 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:33.300940    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:33.300969 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:33.300967    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:33.301212 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:33.301011    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials podName:3d8bd0b9-7dbc-4bab-83d4-18425e9c3575 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:35.300996299 +0000 UTC m=+2538.223454577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:33.654636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:33.654576    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs" event={"ID":"3aa238f7-b340-4e88-b88f-29120c68a52a","Type":"ContainerStarted","Data":"26960ac4fe0b5c4ec3892d8c6d9394201dd18284054ea9cd2fde581a04923fa8"}
Oct 30 21:52:33.655041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:33.654661    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:33.655041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:33.654689    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs" event={"ID":"3aa238f7-b340-4e88-b88f-29120c68a52a","Type":"ContainerStarted","Data":"a3a47ae998da3d5c3c29f9f2b1e3b3f5b72390fa545529b9c9ebdf72f9f70a2f"}
Oct 30 21:52:33.695890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:33.695807    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs" podStartSLOduration=1.695791327 podStartE2EDuration="1.695791327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:33.693921256 +0000 UTC m=+2536.616379553" watchObservedRunningTime="2026-10-30 21:52:33.695791327 +0000 UTC m=+2536.618249625"
Oct 30 21:52:35.346627 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:35.341882    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:35.346627 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:35.341919    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:35.346627 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:35.341983    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials podName:3d8bd0b9-7dbc-4bab-83d4-18425e9c3575 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:39.341964725 +0000 UTC m=+2542.264423004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:37.498572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.498534    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"]
Oct 30 21:52:37.503860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.503829    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.526351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.526315    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"]
Oct 30 21:52:37.531364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.531334    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-90d1425711a8\""
Oct 30 21:52:37.531523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.531380    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"kube-root-ca.crt\""
Oct 30 21:52:37.531523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.531411    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-e2e--68c3d057-dockercfg-zz6jd\""
Oct 30 21:52:37.531523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.531481    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"routing-overlay\""
Oct 30 21:52:37.533899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.533878    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"openshift-service-ca.crt\""
Oct 30 21:52:37.533899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.533883    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ed0866\""
Oct 30 21:52:37.683629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.682281    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwlh\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.683629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.682446    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.683629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.682503    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.683629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.682554    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.783493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.783447    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.783696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.783599    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwlh\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.783775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.783732    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.783872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.783772    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.784237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.784209    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.784616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.784579    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.786181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.786160    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.804308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.804275    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwlh\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:37.820327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:37.820297    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:38.155219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.155189    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"]
Oct 30 21:52:38.157993 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:38.157962    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3765b785_1b74_4e11_b756_b023f8754c99.slice/crio-8665d9e234bf915e926a71a437d5d3be9daefe46826371b31e3f9ba56cc3cfe6 WatchSource:0}: Error finding container 8665d9e234bf915e926a71a437d5d3be9daefe46826371b31e3f9ba56cc3cfe6: Status 404 returned error can't find the container with id 8665d9e234bf915e926a71a437d5d3be9daefe46826371b31e3f9ba56cc3cfe6
Oct 30 21:52:38.347793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.347760    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"]
Oct 30 21:52:38.361919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.361881    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.376020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.375985    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"]
Oct 30 21:52:38.381984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.381953    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:38.381984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.381975    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:52:38.381984 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.381959    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:52:38.382201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.381953    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:52:38.495113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.495019    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.495312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.495176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.495312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.495260    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skclw\" (UniqueName: \"kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.597289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.597245    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.597808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.597391    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.597808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.597503    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skclw\" (UniqueName: \"kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.598228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.598200    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.599872 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.599850    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.619404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.619371    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skclw\" (UniqueName: \"kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw\") pod \"maas-api-revoke-sub-7a6802ba-97xcl\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.678301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.678246    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:38.742763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.742707    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g" event={"ID":"3765b785-1b74-4e11-b756-b023f8754c99","Type":"ContainerStarted","Data":"73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e"}
Oct 30 21:52:38.743002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.742980    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g" event={"ID":"3765b785-1b74-4e11-b756-b023f8754c99","Type":"ContainerStarted","Data":"8665d9e234bf915e926a71a437d5d3be9daefe46826371b31e3f9ba56cc3cfe6"}
Oct 30 21:52:38.743116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.743104    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:38.790653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.788773    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g" podStartSLOduration=1.788754067 podStartE2EDuration="1.788754067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:38.783447819 +0000 UTC m=+2541.705906119" watchObservedRunningTime="2026-10-30 21:52:38.788754067 +0000 UTC m=+2541.711212369"
Oct 30 21:52:38.985106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:38.985072    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"]
Oct 30 21:52:38.988188 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:38.988158    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ad1c09_940d_4adc_b003_ef91008bf5ac.slice/crio-4965ff33ebb2c4bc825ae9f308e202b061a3749ab0bf87b41959ac7abdf6514d WatchSource:0}: Error finding container 4965ff33ebb2c4bc825ae9f308e202b061a3749ab0bf87b41959ac7abdf6514d: Status 404 returned error can't find the container with id 4965ff33ebb2c4bc825ae9f308e202b061a3749ab0bf87b41959ac7abdf6514d
Oct 30 21:52:39.429483 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:39.429449    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:39.429483 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:39.429479    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:39.429739 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:39.429531    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials podName:3d8bd0b9-7dbc-4bab-83d4-18425e9c3575 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:47.429517715 +0000 UTC m=+2550.351975992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:39.754250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:39.754152    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl" event={"ID":"b6ad1c09-940d-4adc-b003-ef91008bf5ac","Type":"ContainerStarted","Data":"165f0192abafab122d2b9a4431ac05a8bd1017e33cb71c03ee93d02c0324bdb0"}
Oct 30 21:52:39.754250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:39.754212    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl" event={"ID":"b6ad1c09-940d-4adc-b003-ef91008bf5ac","Type":"ContainerStarted","Data":"4965ff33ebb2c4bc825ae9f308e202b061a3749ab0bf87b41959ac7abdf6514d"}
Oct 30 21:52:39.800008 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:39.799950    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl" podStartSLOduration=1.799931476 podStartE2EDuration="1.799931476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:39.795676884 +0000 UTC m=+2542.718135184" watchObservedRunningTime="2026-10-30 21:52:39.799931476 +0000 UTC m=+2542.722389945"
Oct 30 21:52:40.764820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:40.764783    6695 generic.go:356] "Generic (PLEG): container finished" podID="b6ad1c09-940d-4adc-b003-ef91008bf5ac" containerID="165f0192abafab122d2b9a4431ac05a8bd1017e33cb71c03ee93d02c0324bdb0" exitCode=0
Oct 30 21:52:40.765264 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:40.764856    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl" event={"ID":"b6ad1c09-940d-4adc-b003-ef91008bf5ac","Type":"ContainerDied","Data":"165f0192abafab122d2b9a4431ac05a8bd1017e33cb71c03ee93d02c0324bdb0"}
Oct 30 21:52:41.972314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:41.972286    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:42.071112 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.071023    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca\") pod \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") "
Oct 30 21:52:42.071286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.071130    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca\") pod \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") "
Oct 30 21:52:42.071286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.071192    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw\" (UniqueName: \"kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw\") pod \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\" (UID: \"b6ad1c09-940d-4adc-b003-ef91008bf5ac\") "
Oct 30 21:52:42.071533 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.071503    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca" pod "b6ad1c09-940d-4adc-b003-ef91008bf5ac" (UID: "b6ad1c09-940d-4adc-b003-ef91008bf5ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:42.071723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.071706    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:42.073310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.073285    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca" pod "b6ad1c09-940d-4adc-b003-ef91008bf5ac" (UID: "b6ad1c09-940d-4adc-b003-ef91008bf5ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:42.073310 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.073299    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw" pod "b6ad1c09-940d-4adc-b003-ef91008bf5ac" (UID: "b6ad1c09-940d-4adc-b003-ef91008bf5ac"). InnerVolumeSpecName "kube-api-access-skclw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:42.172672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.172640    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ad1c09-940d-4adc-b003-ef91008bf5ac-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:42.172672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.172669    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skclw\" (UniqueName: \"kubernetes.io/projected/b6ad1c09-940d-4adc-b003-ef91008bf5ac-kube-api-access-skclw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:42.647461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.647431    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:42.820839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.820804    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"
Oct 30 21:52:42.821045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.820842    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl" event={"ID":"b6ad1c09-940d-4adc-b003-ef91008bf5ac","Type":"ContainerDied","Data":"4965ff33ebb2c4bc825ae9f308e202b061a3749ab0bf87b41959ac7abdf6514d"}
Oct 30 21:52:42.821045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:42.820934    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4965ff33ebb2c4bc825ae9f308e202b061a3749ab0bf87b41959ac7abdf6514d"
Oct 30 21:52:44.688627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:44.683801    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:52:44.777626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:44.774193    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"]
Oct 30 21:52:46.876273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.876235    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"]
Oct 30 21:52:46.877160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.877131    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b6ad1c09-940d-4adc-b003-ef91008bf5ac" containerName="revoke-keys"
Oct 30 21:52:46.877231 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.877161    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ad1c09-940d-4adc-b003-ef91008bf5ac" containerName="revoke-keys"
Oct 30 21:52:46.877289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.877280    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ad1c09-940d-4adc-b003-ef91008bf5ac" containerName="revoke-keys"
Oct 30 21:52:46.881056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.881035    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:46.906964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.906933    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"kube-root-ca.crt\""
Oct 30 21:52:46.907155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.907011    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-plugins-e2e-extmod-b-db1288\""
Oct 30 21:52:46.907155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.907013    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"openshift-service-ca.crt\""
Oct 30 21:52:46.907155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.907049    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"routing-overlay\""
Oct 30 21:52:46.909572 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.909542    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9gkx\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:46.909753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.909718    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:46.909834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.909807    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:46.909899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.909840    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:46.911982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.911958    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"]
Oct 30 21:52:46.914763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.914739    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-44df4141bbb0\""
Oct 30 21:52:46.914979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.914892    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-c26bbe9ee934\""
Oct 30 21:52:46.914979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:46.914959    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-e2e--f0c0883f-dockercfg-8mgn7\""
Oct 30 21:52:47.011236 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.011192    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9gkx\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.011406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.011294    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.011406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.011359    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.011406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.011383    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.012324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.012286    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.012483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.012440    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.014157 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.014134    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.030750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.030718    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9gkx\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.195732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.195639    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.526417 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:47.526374    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:47.526417 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:47.526415    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:47.526696 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:47.526486    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials podName:3d8bd0b9-7dbc-4bab-83d4-18425e9c3575 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:03.526469061 +0000 UTC m=+2566.448927362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:52:47.666019 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.665989    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"]
Oct 30 21:52:47.669680 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:52:47.669651    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9015de0_5148_4d58_8dec_4d1316b3e9cf.slice/crio-e57893f14986bd41d3a17a8bc5f8d3b4e135e431b21a6f08c79d9f4fe23daedd WatchSource:0}: Error finding container e57893f14986bd41d3a17a8bc5f8d3b4e135e431b21a6f08c79d9f4fe23daedd: Status 404 returned error can't find the container with id e57893f14986bd41d3a17a8bc5f8d3b4e135e431b21a6f08c79d9f4fe23daedd
Oct 30 21:52:47.871627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.871510    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb" event={"ID":"c9015de0-5148-4d58-8dec-4d1316b3e9cf","Type":"ContainerStarted","Data":"280564797d3f30902b515df658e3c48a76c969abcfc8a9677751db2e2dd0399a"}
Oct 30 21:52:47.871627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.871567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb" event={"ID":"c9015de0-5148-4d58-8dec-4d1316b3e9cf","Type":"ContainerStarted","Data":"e57893f14986bd41d3a17a8bc5f8d3b4e135e431b21a6f08c79d9f4fe23daedd"}
Oct 30 21:52:47.871852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.871641    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:52:47.922882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:47.922817    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb" podStartSLOduration=1.922797042 podStartE2EDuration="1.922797042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:47.921149883 +0000 UTC m=+2550.843608181" watchObservedRunningTime="2026-10-30 21:52:47.922797042 +0000 UTC m=+2550.845255343"
Oct 30 21:52:49.758016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.757980    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:52:49.774768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.774734    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" podUID="3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" containerName="payload-processing" containerID="cri-o://bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe" gracePeriod=25
Oct 30 21:52:49.878862 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.878820    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"]
Oct 30 21:52:49.884471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.884445    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:49.902428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.902405    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"agc-cred-6062e2cf285d\""
Oct 30 21:52:49.917393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.917346    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"]
Oct 30 21:52:49.975911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.975876    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrnl\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:49.976082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.975974    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:49.976471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.976436    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:49.976630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:49.976489    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.078161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.078130    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.078340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.078170    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.078340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.078218    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrnl\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.078340 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.078273    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.078619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.078584    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:50.078914 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.078891    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.079058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.079035    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.081173 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.081153    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.097188 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.097157    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrnl\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.179095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.179062    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf\") pod \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") "
Oct 30 21:52:50.179269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.179114    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials\") pod \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") "
Oct 30 21:52:50.179269 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.179221    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume\") pod \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") "
Oct 30 21:52:50.179408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.179272    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume\") pod \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\" (UID: \"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575\") "
Oct 30 21:52:50.179764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.179725    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume" pod "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:50.179764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.179730    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume" pod "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:50.180064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.180045    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:50.180120 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.180069    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:50.181577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.181558    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf" pod "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575"). InnerVolumeSpecName "kube-api-access-sg4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:50.181659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.181587    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials" pod "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" (UID: "3d8bd0b9-7dbc-4bab-83d4-18425e9c3575"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:50.200597 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.200567    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.282655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.281634    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4cf\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-kube-api-access-sg4cf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:50.282655 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.281671    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:50.383056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.382969    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"]
Oct 30 21:52:50.618626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.602380    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"]
Oct 30 21:52:50.908675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.906596    6695 generic.go:356] "Generic (PLEG): container finished" podID="3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" containerID="bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe" exitCode=0
Oct 30 21:52:50.908675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.906788    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" event={"ID":"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575","Type":"ContainerDied","Data":"bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe"}
Oct 30 21:52:50.908675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.906870    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6" event={"ID":"3d8bd0b9-7dbc-4bab-83d4-18425e9c3575","Type":"ContainerDied","Data":"dc725af8f53ec9b220b2968acff4aa07f2b1a71719fe72817ba8a284d7d185f7"}
Oct 30 21:52:50.908675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.906897    6695 scope.go:122] "RemoveContainer" containerID="bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe"
Oct 30 21:52:50.908675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.907070    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"
Oct 30 21:52:50.927285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.927060    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm" event={"ID":"7e2483d3-3a33-4dcd-b578-838b2887abd4","Type":"ContainerStarted","Data":"e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d"}
Oct 30 21:52:50.927285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.927133    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm" event={"ID":"7e2483d3-3a33-4dcd-b578-838b2887abd4","Type":"ContainerStarted","Data":"a17fa6276a7626b15f6f3b1e2928a25c9a9ef49801e99cf4dbe51660e0f387f7"}
Oct 30 21:52:50.928314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.928253    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:52:50.949526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.949165    6695 scope.go:122] "RemoveContainer" containerID="bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe"
Oct 30 21:52:50.950216 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:50.950106    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe\": container with ID starting with bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe not found: ID does not exist" containerID="bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe"
Oct 30 21:52:50.950216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.950147    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe"} err="failed to get container status \"bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe\": rpc error: code = NotFound desc = could not find container \"bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe\": container with ID starting with bbc2ffe20eeab38660aa065dc8138e503e99d911e25f87f039fcb5d1f59f17fe not found: ID does not exist"
Oct 30 21:52:50.989518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:50.987111    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm" podStartSLOduration=1.987091301 podStartE2EDuration="1.987091301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:50.985042852 +0000 UTC m=+2553.907501151" watchObservedRunningTime="2026-10-30 21:52:50.987091301 +0000 UTC m=+2553.909549599"
Oct 30 21:52:51.029270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:51.029232    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"]
Oct 30 21:52:51.037890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:51.037853    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774b999b6"]
Oct 30 21:52:51.217232 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:51.217137    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" path="/var/lib/kubelet/pods/3d8bd0b9-7dbc-4bab-83d4-18425e9c3575/volumes"
Oct 30 21:52:55.384569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.384524    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v" podUID="55189435-f54b-4dc6-bb85-11878561a1b9" containerName="payload-processing" containerID="cri-o://ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac" gracePeriod=25
Oct 30 21:52:55.662257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.662228    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:55.791564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.791523    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume\") pod \"55189435-f54b-4dc6-bb85-11878561a1b9\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") "
Oct 30 21:52:55.791788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.791659    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials\") pod \"55189435-f54b-4dc6-bb85-11878561a1b9\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") "
Oct 30 21:52:55.791788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.791726    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume\") pod \"55189435-f54b-4dc6-bb85-11878561a1b9\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") "
Oct 30 21:52:55.791788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.791752    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8\") pod \"55189435-f54b-4dc6-bb85-11878561a1b9\" (UID: \"55189435-f54b-4dc6-bb85-11878561a1b9\") "
Oct 30 21:52:55.792064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.792030    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume" pod "55189435-f54b-4dc6-bb85-11878561a1b9" (UID: "55189435-f54b-4dc6-bb85-11878561a1b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:55.792208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.792183    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume" pod "55189435-f54b-4dc6-bb85-11878561a1b9" (UID: "55189435-f54b-4dc6-bb85-11878561a1b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:55.792565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.792545    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:55.792565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.792562    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55189435-f54b-4dc6-bb85-11878561a1b9-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:55.793880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.793851    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials" pod "55189435-f54b-4dc6-bb85-11878561a1b9" (UID: "55189435-f54b-4dc6-bb85-11878561a1b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:55.794022 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.793997    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8" pod "55189435-f54b-4dc6-bb85-11878561a1b9" (UID: "55189435-f54b-4dc6-bb85-11878561a1b9"). InnerVolumeSpecName "kube-api-access-flls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:55.893962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.893867    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:55.893962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.893902    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flls8\" (UniqueName: \"kubernetes.io/projected/55189435-f54b-4dc6-bb85-11878561a1b9-kube-api-access-flls8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:52:55.969416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.969376    6695 generic.go:356] "Generic (PLEG): container finished" podID="55189435-f54b-4dc6-bb85-11878561a1b9" containerID="ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac" exitCode=0
Oct 30 21:52:55.969631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.969444    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v" event={"ID":"55189435-f54b-4dc6-bb85-11878561a1b9","Type":"ContainerDied","Data":"ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac"}
Oct 30 21:52:55.969631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.969505    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"
Oct 30 21:52:55.969631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.969520    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v" event={"ID":"55189435-f54b-4dc6-bb85-11878561a1b9","Type":"ContainerDied","Data":"07ff7c815a1acf93f1a25e6d6d98cb56641cb01e721dd1e993cbeefa9acbf3bd"}
Oct 30 21:52:55.969631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.969550    6695 scope.go:122] "RemoveContainer" containerID="ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac"
Oct 30 21:52:55.983382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.983360    6695 scope.go:122] "RemoveContainer" containerID="ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac"
Oct 30 21:52:55.983781 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:52:55.983745    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac\": container with ID starting with ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac not found: ID does not exist" containerID="ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac"
Oct 30 21:52:55.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:55.983783    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac"} err="failed to get container status \"ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac\": rpc error: code = NotFound desc = could not find container \"ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac\": container with ID starting with ae3999b7fe73f0670d0ba6acc4ba52c9b97a7efc61ca6d5b6bdaccd2270a8eac not found: ID does not exist"
Oct 30 21:52:56.026902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:56.026870    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"]
Oct 30 21:52:56.037442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:56.037411    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-79cf6qr4v"]
Oct 30 21:52:57.216375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:57.216341    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55189435-f54b-4dc6-bb85-11878561a1b9" path="/var/lib/kubelet/pods/55189435-f54b-4dc6-bb85-11878561a1b9/volumes"
Oct 30 21:52:58.885305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:52:58.885274    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:53:01.077309 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:53:01.077266    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:53:01.939764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:01.939733    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:53:02.010818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:02.010776    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"]
Oct 30 21:53:05.319568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.319526    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"]
Oct 30 21:53:05.320577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.320550    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" containerName="payload-processing"
Oct 30 21:53:05.320577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.320578    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" containerName="payload-processing"
Oct 30 21:53:05.320811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.320614    6695 cpu_manager.go:414] "containerMap: removing container" podUID="55189435-f54b-4dc6-bb85-11878561a1b9" containerName="payload-processing"
Oct 30 21:53:05.320811 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.320623    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55189435-f54b-4dc6-bb85-11878561a1b9" containerName="payload-processing"
Oct 30 21:53:05.320926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.320849    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8bd0b9-7dbc-4bab-83d4-18425e9c3575" containerName="payload-processing"
Oct 30 21:53:05.320926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.320872    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="55189435-f54b-4dc6-bb85-11878561a1b9" containerName="payload-processing"
Oct 30 21:53:05.326404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.326380    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.352939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.352900    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"e2e-cmp-legacy-6fcb19-provider-cred\""
Oct 30 21:53:05.352939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.352922    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 21:53:05.352939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.352905    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 21:53:05.353233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.352986    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 21:53:05.353233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.352993    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-vs2sd\""
Oct 30 21:53:05.353233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.353076    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 21:53:05.357108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.357080    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"]
Oct 30 21:53:05.431227 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.431189    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.431436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.431333    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.431436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.431402    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.431567 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.431547    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2mr\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.534625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.534317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2mr\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.534625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.534510    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.534625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.534622    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.534987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.534679    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.557624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.535506    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.557624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.536583    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.558244 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.558208    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.591828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.591746    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2mr\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.641898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.641863    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:05.962561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:05.962535    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"]
Oct 30 21:53:05.965978 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:53:05.965943    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15548e1b_fc98_4ba4_8a15_d2cc85159602.slice/crio-a1eb05ede8c326d74b5f48527e7168606047eb41b8b0eda23e7f02a6e432c39f WatchSource:0}: Error finding container a1eb05ede8c326d74b5f48527e7168606047eb41b8b0eda23e7f02a6e432c39f: Status 404 returned error can't find the container with id a1eb05ede8c326d74b5f48527e7168606047eb41b8b0eda23e7f02a6e432c39f
Oct 30 21:53:06.149636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:06.149566    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq" event={"ID":"15548e1b-fc98-4ba4-8a15-d2cc85159602","Type":"ContainerStarted","Data":"9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8"}
Oct 30 21:53:06.149636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:06.149635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq" event={"ID":"15548e1b-fc98-4ba4-8a15-d2cc85159602","Type":"ContainerStarted","Data":"a1eb05ede8c326d74b5f48527e7168606047eb41b8b0eda23e7f02a6e432c39f"}
Oct 30 21:53:06.149885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:06.149682    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:06.189765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:06.189707    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq" podStartSLOduration=1.189693855 podStartE2EDuration="1.189693855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:06.188995196 +0000 UTC m=+2569.111453494" watchObservedRunningTime="2026-10-30 21:53:06.189693855 +0000 UTC m=+2569.112152147"
Oct 30 21:53:07.012100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.012044    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7" podUID="4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" containerName="payload-processing" containerID="cri-o://36797815398d50890e48a3e98540a6d0f88a4a7a71b636f9d71819fbf3c983b1" gracePeriod=25
Oct 30 21:53:07.161517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.161474    6695 generic.go:356] "Generic (PLEG): container finished" podID="4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" containerID="36797815398d50890e48a3e98540a6d0f88a4a7a71b636f9d71819fbf3c983b1" exitCode=0
Oct 30 21:53:07.161908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.161868    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7" event={"ID":"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9","Type":"ContainerDied","Data":"36797815398d50890e48a3e98540a6d0f88a4a7a71b636f9d71819fbf3c983b1"}
Oct 30 21:53:07.295262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.295235    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:53:07.361982 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.361949    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp\") pod \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") "
Oct 30 21:53:07.362249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362010    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume\") pod \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") "
Oct 30 21:53:07.362249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362092    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume\") pod \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") "
Oct 30 21:53:07.362249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362186    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials\") pod \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\" (UID: \"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9\") "
Oct 30 21:53:07.362489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362462    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume" pod "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" (UID: "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:07.362559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362521    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume" pod "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" (UID: "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:07.362856 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362839    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:07.362926 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.362857    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:07.364818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.364796    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp" pod "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" (UID: "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9"). InnerVolumeSpecName "kube-api-access-4zstp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:07.364929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.364902    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials" pod "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" (UID: "4c16d6f7-7fc5-4c1c-a824-07ba261aeda9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:07.464233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.464190    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:07.464233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:07.464228    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zstp\" (UniqueName: \"kubernetes.io/projected/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9-kube-api-access-4zstp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:08.171955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:08.171920    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"
Oct 30 21:53:08.171955 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:08.171942    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7" event={"ID":"4c16d6f7-7fc5-4c1c-a824-07ba261aeda9","Type":"ContainerDied","Data":"798cb8da979fea8b19a26c04e44a556bd4a6fb6b78115f9c37af3267278753b4"}
Oct 30 21:53:08.172512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:08.172037    6695 scope.go:122] "RemoveContainer" containerID="36797815398d50890e48a3e98540a6d0f88a4a7a71b636f9d71819fbf3c983b1"
Oct 30 21:53:08.223531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:08.223491    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"]
Oct 30 21:53:08.234083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:08.233991    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67mtmk7"]
Oct 30 21:53:09.215916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:09.215883    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" path="/var/lib/kubelet/pods/4c16d6f7-7fc5-4c1c-a824-07ba261aeda9/volumes"
Oct 30 21:53:17.165080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:17.165045    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:53:20.766749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.766715    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"]
Oct 30 21:53:20.767560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.767545    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" containerName="payload-processing"
Oct 30 21:53:20.767662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.767561    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" containerName="payload-processing"
Oct 30 21:53:20.767712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.767690    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c16d6f7-7fc5-4c1c-a824-07ba261aeda9" containerName="payload-processing"
Oct 30 21:53:20.771755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.771732    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.793499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.793461    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:53:20.793499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.793469    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:53:20.799614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.799557    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"]
Oct 30 21:53:20.805068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.805038    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:20.805247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.805072    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:53:20.885521 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.885483    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbjg\" (UniqueName: \"kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.885720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.885558    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.885774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.885752    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.987150 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.987109    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbjg\" (UniqueName: \"kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.987353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.987186    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.987353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.987248    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.988032 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.987999    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:20.989848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:20.989821    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:21.013172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:21.013140    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbjg\" (UniqueName: \"kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg\") pod \"maas-api-revoke-sub-8f432bbf-m9hlj\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:21.087686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:21.087644    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:21.421417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:21.421387    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"]
Oct 30 21:53:21.424250 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:53:21.424223    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bdee61_9987_408e_b85c_54375f857b17.slice/crio-c495d0761243944477e3d9151b476203dc7707f8984df9a03453d0e75f20a5c2 WatchSource:0}: Error finding container c495d0761243944477e3d9151b476203dc7707f8984df9a03453d0e75f20a5c2: Status 404 returned error can't find the container with id c495d0761243944477e3d9151b476203dc7707f8984df9a03453d0e75f20a5c2
Oct 30 21:53:22.376119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.376088    6695 scope.go:122] "RemoveContainer" containerID="7aed659b9721f7eb4fe5c615b26cca74b413093a2513c128a356cad8d434fc1d"
Oct 30 21:53:22.396746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.396720    6695 scope.go:122] "RemoveContainer" containerID="a27073e776c294281dd4b37c7af202ad43877923a45b119ae033ef0ec356ab36"
Oct 30 21:53:22.408160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.408130    6695 scope.go:122] "RemoveContainer" containerID="448bf8814b33d29959b89e224f83c12ec402e76ceb4746b32836031e9bd84d87"
Oct 30 21:53:22.418587 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.418563    6695 scope.go:122] "RemoveContainer" containerID="e311d0e41d82d378f79a93181521ce59060cb8deee4466247ae0cad14a51abc7"
Oct 30 21:53:22.420773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.420746    6695 generic.go:356] "Generic (PLEG): container finished" podID="f7bdee61-9987-408e-b85c-54375f857b17" containerID="7703d30c1cac5a70283bee38aad814c55dc4b7773a480cebf5dc6be35d8cf755" exitCode=0
Oct 30 21:53:22.420895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.420785    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj" event={"ID":"f7bdee61-9987-408e-b85c-54375f857b17","Type":"ContainerDied","Data":"7703d30c1cac5a70283bee38aad814c55dc4b7773a480cebf5dc6be35d8cf755"}
Oct 30 21:53:22.420895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.420858    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj" event={"ID":"f7bdee61-9987-408e-b85c-54375f857b17","Type":"ContainerStarted","Data":"c495d0761243944477e3d9151b476203dc7707f8984df9a03453d0e75f20a5c2"}
Oct 30 21:53:22.428514 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.428497    6695 scope.go:122] "RemoveContainer" containerID="a4bd05406baf0e46e6adee1e7109cb02f5fdada9d361dc7067030066f382b797"
Oct 30 21:53:22.438040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:22.438021    6695 scope.go:122] "RemoveContainer" containerID="cb15c9cb87f2ef896e8502880ac61605c7e9b28bad5aca014487a7f3e5cad14e"
Oct 30 21:53:23.615535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.615506    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:23.728109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.728018    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg\" (UniqueName: \"kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg\") pod \"f7bdee61-9987-408e-b85c-54375f857b17\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") "
Oct 30 21:53:23.728303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.728141    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca\") pod \"f7bdee61-9987-408e-b85c-54375f857b17\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") "
Oct 30 21:53:23.728303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.728196    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca\") pod \"f7bdee61-9987-408e-b85c-54375f857b17\" (UID: \"f7bdee61-9987-408e-b85c-54375f857b17\") "
Oct 30 21:53:23.728584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.728560    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca" pod "f7bdee61-9987-408e-b85c-54375f857b17" (UID: "f7bdee61-9987-408e-b85c-54375f857b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:23.728845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.728823    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:23.730296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.730269    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca" pod "f7bdee61-9987-408e-b85c-54375f857b17" (UID: "f7bdee61-9987-408e-b85c-54375f857b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:23.730356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.730286    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg" pod "f7bdee61-9987-408e-b85c-54375f857b17" (UID: "f7bdee61-9987-408e-b85c-54375f857b17"). InnerVolumeSpecName "kube-api-access-bpbjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:23.830679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.830644    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbjg\" (UniqueName: \"kubernetes.io/projected/f7bdee61-9987-408e-b85c-54375f857b17-kube-api-access-bpbjg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:23.830679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:23.830676    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7bdee61-9987-408e-b85c-54375f857b17-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:24.218507 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:24.218469    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"]
Oct 30 21:53:24.438384 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:24.438355    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"
Oct 30 21:53:24.438566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:24.438392    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj" event={"ID":"f7bdee61-9987-408e-b85c-54375f857b17","Type":"ContainerDied","Data":"c495d0761243944477e3d9151b476203dc7707f8984df9a03453d0e75f20a5c2"}
Oct 30 21:53:24.438566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:24.438441    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c495d0761243944477e3d9151b476203dc7707f8984df9a03453d0e75f20a5c2"
Oct 30 21:53:29.218827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.218787    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb" podUID="c9015de0-5148-4d58-8dec-4d1316b3e9cf" containerName="payload-processing" containerID="cri-o://280564797d3f30902b515df658e3c48a76c969abcfc8a9677751db2e2dd0399a" gracePeriod=25
Oct 30 21:53:29.498593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.498500    6695 generic.go:356] "Generic (PLEG): container finished" podID="c9015de0-5148-4d58-8dec-4d1316b3e9cf" containerID="280564797d3f30902b515df658e3c48a76c969abcfc8a9677751db2e2dd0399a" exitCode=0
Oct 30 21:53:29.498593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.498572    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb" event={"ID":"c9015de0-5148-4d58-8dec-4d1316b3e9cf","Type":"ContainerDied","Data":"280564797d3f30902b515df658e3c48a76c969abcfc8a9677751db2e2dd0399a"}
Oct 30 21:53:29.542481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.542455    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:53:29.714684 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.714616    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume\") pod \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") "
Oct 30 21:53:29.715011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.714988    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume\") pod \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") "
Oct 30 21:53:29.715356 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.715335    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials\") pod \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") "
Oct 30 21:53:29.715650 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.715595    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx\") pod \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\" (UID: \"c9015de0-5148-4d58-8dec-4d1316b3e9cf\") "
Oct 30 21:53:29.719485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.719272    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx" pod "c9015de0-5148-4d58-8dec-4d1316b3e9cf" (UID: "c9015de0-5148-4d58-8dec-4d1316b3e9cf"). InnerVolumeSpecName "kube-api-access-j9gkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.720092 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.719829    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume" pod "c9015de0-5148-4d58-8dec-4d1316b3e9cf" (UID: "c9015de0-5148-4d58-8dec-4d1316b3e9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:29.720656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.720400    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume" pod "c9015de0-5148-4d58-8dec-4d1316b3e9cf" (UID: "c9015de0-5148-4d58-8dec-4d1316b3e9cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:29.722562 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.722536    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials" pod "c9015de0-5148-4d58-8dec-4d1316b3e9cf" (UID: "c9015de0-5148-4d58-8dec-4d1316b3e9cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:29.818139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.818090    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:29.818139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.818128    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9gkx\" (UniqueName: \"kubernetes.io/projected/c9015de0-5148-4d58-8dec-4d1316b3e9cf-kube-api-access-j9gkx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:29.818139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.818143    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:29.818427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:29.818154    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9015de0-5148-4d58-8dec-4d1316b3e9cf-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:30.507887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:30.507853    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"
Oct 30 21:53:30.508404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:30.507864    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb" event={"ID":"c9015de0-5148-4d58-8dec-4d1316b3e9cf","Type":"ContainerDied","Data":"e57893f14986bd41d3a17a8bc5f8d3b4e135e431b21a6f08c79d9f4fe23daedd"}
Oct 30 21:53:30.508404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:30.507988    6695 scope.go:122] "RemoveContainer" containerID="280564797d3f30902b515df658e3c48a76c969abcfc8a9677751db2e2dd0399a"
Oct 30 21:53:30.562808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:30.562777    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"]
Oct 30 21:53:30.573632 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:30.573574    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6988l8vcb"]
Oct 30 21:53:31.076100 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:53:31.076065    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:53:31.216027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.215991    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9015de0-5148-4d58-8dec-4d1316b3e9cf" path="/var/lib/kubelet/pods/c9015de0-5148-4d58-8dec-4d1316b3e9cf/volumes"
Oct 30 21:53:31.349505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.349431    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"]
Oct 30 21:53:31.350258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.350242    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c9015de0-5148-4d58-8dec-4d1316b3e9cf" containerName="payload-processing"
Oct 30 21:53:31.350311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.350260    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9015de0-5148-4d58-8dec-4d1316b3e9cf" containerName="payload-processing"
Oct 30 21:53:31.350311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.350286    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f7bdee61-9987-408e-b85c-54375f857b17" containerName="revoke-keys"
Oct 30 21:53:31.350311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.350291    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bdee61-9987-408e-b85c-54375f857b17" containerName="revoke-keys"
Oct 30 21:53:31.350430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.350421    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9015de0-5148-4d58-8dec-4d1316b3e9cf" containerName="payload-processing"
Oct 30 21:53:31.350466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.350437    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bdee61-9987-408e-b85c-54375f857b17" containerName="revoke-keys"
Oct 30 21:53:31.355610 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.355586    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.373327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.373299    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:31.373446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.373370    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:53:31.377793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.377771    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:53:31.377793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.377778    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:53:31.401184 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.401150    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"]
Oct 30 21:53:31.440938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.440889    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.441165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.441050    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.441165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.441093    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfzr\" (UniqueName: \"kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.542761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.542724    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.543228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.542781    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfzr\" (UniqueName: \"kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.543228 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.542944    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.543427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.543402    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.545344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.545321    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.561870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.561840    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfzr\" (UniqueName: \"kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr\") pod \"maas-api-revoke-sub-ef9c476b-8pjkk\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.671054 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.670967    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:31.979773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:31.979743    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"]
Oct 30 21:53:31.982545 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:53:31.982508    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f97aec_fd57_404a_b15c_e024c11a6c78.slice/crio-4a578d7bd8e958f5a26eeca5c75a5b1fa87d023046300fdcd98794415ae2fc2a WatchSource:0}: Error finding container 4a578d7bd8e958f5a26eeca5c75a5b1fa87d023046300fdcd98794415ae2fc2a: Status 404 returned error can't find the container with id 4a578d7bd8e958f5a26eeca5c75a5b1fa87d023046300fdcd98794415ae2fc2a
Oct 30 21:53:32.525210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:32.525177    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk" event={"ID":"b4f97aec-fd57-404a-b15c-e024c11a6c78","Type":"ContainerStarted","Data":"117b32de4501f3b36920692bac35a16e3a4746787f71843b68069ba781a5a1c9"}
Oct 30 21:53:32.525401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:32.525231    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk" event={"ID":"b4f97aec-fd57-404a-b15c-e024c11a6c78","Type":"ContainerStarted","Data":"4a578d7bd8e958f5a26eeca5c75a5b1fa87d023046300fdcd98794415ae2fc2a"}
Oct 30 21:53:32.563592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:32.563527    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk" podStartSLOduration=1.563509297 podStartE2EDuration="1.563509297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:32.56148612 +0000 UTC m=+2595.483944418" watchObservedRunningTime="2026-10-30 21:53:32.563509297 +0000 UTC m=+2595.485967589"
Oct 30 21:53:33.534781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:33.534742    6695 generic.go:356] "Generic (PLEG): container finished" podID="b4f97aec-fd57-404a-b15c-e024c11a6c78" containerID="117b32de4501f3b36920692bac35a16e3a4746787f71843b68069ba781a5a1c9" exitCode=0
Oct 30 21:53:33.534946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:33.534810    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk" event={"ID":"b4f97aec-fd57-404a-b15c-e024c11a6c78","Type":"ContainerDied","Data":"117b32de4501f3b36920692bac35a16e3a4746787f71843b68069ba781a5a1c9"}
Oct 30 21:53:34.737001 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.736971    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:34.789553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.789518    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca\") pod \"b4f97aec-fd57-404a-b15c-e024c11a6c78\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") "
Oct 30 21:53:34.789835 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.789574    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca\") pod \"b4f97aec-fd57-404a-b15c-e024c11a6c78\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") "
Oct 30 21:53:34.790463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.790436    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca" pod "b4f97aec-fd57-404a-b15c-e024c11a6c78" (UID: "b4f97aec-fd57-404a-b15c-e024c11a6c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:34.792776 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.792669    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca" pod "b4f97aec-fd57-404a-b15c-e024c11a6c78" (UID: "b4f97aec-fd57-404a-b15c-e024c11a6c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:34.890927 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.890892    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr\" (UniqueName: \"kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr\") pod \"b4f97aec-fd57-404a-b15c-e024c11a6c78\" (UID: \"b4f97aec-fd57-404a-b15c-e024c11a6c78\") "
Oct 30 21:53:34.891238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.891222    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:34.891319 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.891242    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4f97aec-fd57-404a-b15c-e024c11a6c78-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:34.893156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.893130    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr" pod "b4f97aec-fd57-404a-b15c-e024c11a6c78" (UID: "b4f97aec-fd57-404a-b15c-e024c11a6c78"). InnerVolumeSpecName "kube-api-access-ncfzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:34.992414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:34.992374    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncfzr\" (UniqueName: \"kubernetes.io/projected/b4f97aec-fd57-404a-b15c-e024c11a6c78-kube-api-access-ncfzr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:35.552637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:35.552589    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk" event={"ID":"b4f97aec-fd57-404a-b15c-e024c11a6c78","Type":"ContainerDied","Data":"4a578d7bd8e958f5a26eeca5c75a5b1fa87d023046300fdcd98794415ae2fc2a"}
Oct 30 21:53:35.552637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:35.552643    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"
Oct 30 21:53:35.552920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:35.552655    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a578d7bd8e958f5a26eeca5c75a5b1fa87d023046300fdcd98794415ae2fc2a"
Oct 30 21:53:44.862671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.862629    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"]
Oct 30 21:53:44.863631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.863598    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b4f97aec-fd57-404a-b15c-e024c11a6c78" containerName="revoke-keys"
Oct 30 21:53:44.863631 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.863631    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f97aec-fd57-404a-b15c-e024c11a6c78" containerName="revoke-keys"
Oct 30 21:53:44.863787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.863777    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f97aec-fd57-404a-b15c-e024c11a6c78" containerName="revoke-keys"
Oct 30 21:53:44.868107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.868084    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:44.887140 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.887102    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"]
Oct 30 21:53:44.893464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.893427    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:44.893640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.893475    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:53:44.893640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.893550    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:53:44.894093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.894070    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:53:44.931145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.931106    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:44.931458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.931427    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:44.931653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:44.931462    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fn6k\" (UniqueName: \"kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.032756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.032722    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.032952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.032918    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.032952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.032936    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fn6k\" (UniqueName: \"kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.033528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.033499    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.035344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.035324    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.051665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.051633    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fn6k\" (UniqueName: \"kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k\") pod \"maas-api-revoke-sub-f595711d-v854n\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.183363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.183266    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:45.490999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.490968    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"]
Oct 30 21:53:45.645807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:45.645773    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n" event={"ID":"5571c25c-dbf5-45e5-911f-18610ca5469d","Type":"ContainerStarted","Data":"d89447019bd4d9c752f48c7275d8cdf434cea36f5081711306652799fdd3ab20"}
Oct 30 21:53:46.655757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:46.655716    6695 generic.go:356] "Generic (PLEG): container finished" podID="5571c25c-dbf5-45e5-911f-18610ca5469d" containerID="14683120529b77122bd39fe7ec1971ac78e0679dd09f73c5ba0e24d3ecb6f795" exitCode=0
Oct 30 21:53:46.656214 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:46.655802    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n" event={"ID":"5571c25c-dbf5-45e5-911f-18610ca5469d","Type":"ContainerDied","Data":"14683120529b77122bd39fe7ec1971ac78e0679dd09f73c5ba0e24d3ecb6f795"}
Oct 30 21:53:47.859295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.859268    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:53:47.974577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.974493    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k\" (UniqueName: \"kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k\") pod \"5571c25c-dbf5-45e5-911f-18610ca5469d\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") "
Oct 30 21:53:47.974764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.974641    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca\") pod \"5571c25c-dbf5-45e5-911f-18610ca5469d\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") "
Oct 30 21:53:47.974819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.974802    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca\") pod \"5571c25c-dbf5-45e5-911f-18610ca5469d\" (UID: \"5571c25c-dbf5-45e5-911f-18610ca5469d\") "
Oct 30 21:53:47.975134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.975102    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca" pod "5571c25c-dbf5-45e5-911f-18610ca5469d" (UID: "5571c25c-dbf5-45e5-911f-18610ca5469d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:47.975430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.975411    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:47.976752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.976719    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k" pod "5571c25c-dbf5-45e5-911f-18610ca5469d" (UID: "5571c25c-dbf5-45e5-911f-18610ca5469d"). InnerVolumeSpecName "kube-api-access-6fn6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:47.976866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:47.976783    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca" pod "5571c25c-dbf5-45e5-911f-18610ca5469d" (UID: "5571c25c-dbf5-45e5-911f-18610ca5469d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:48.076990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:48.076952    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fn6k\" (UniqueName: \"kubernetes.io/projected/5571c25c-dbf5-45e5-911f-18610ca5469d-kube-api-access-6fn6k\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:48.076990 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:48.076986    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5571c25c-dbf5-45e5-911f-18610ca5469d-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:53:48.673568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:48.673532    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n" event={"ID":"5571c25c-dbf5-45e5-911f-18610ca5469d","Type":"ContainerDied","Data":"d89447019bd4d9c752f48c7275d8cdf434cea36f5081711306652799fdd3ab20"}
Oct 30 21:53:48.673769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:48.673582    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89447019bd4d9c752f48c7275d8cdf434cea36f5081711306652799fdd3ab20"
Oct 30 21:53:48.673769 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:53:48.673536    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"
Oct 30 21:54:01.076014 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:54:01.075924    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:54:10.010149 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.010113    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"]
Oct 30 21:54:10.011030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.011012    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5571c25c-dbf5-45e5-911f-18610ca5469d" containerName="revoke-keys"
Oct 30 21:54:10.011111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.011032    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5571c25c-dbf5-45e5-911f-18610ca5469d" containerName="revoke-keys"
Oct 30 21:54:10.011179 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.011170    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5571c25c-dbf5-45e5-911f-18610ca5469d" containerName="revoke-keys"
Oct 30 21:54:10.015394 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.015371    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.040923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.040887    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.041125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.040939    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5stf\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.041125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.041063    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.041125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.041107    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.050474 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.050437    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"]
Oct 30 21:54:10.142463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.142428    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.142786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.142479    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5stf\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.142786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.142531    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.142786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.142554    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.143503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.143480    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.143579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.143537    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.145196 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.145176    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.162723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.162697    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5stf\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.331382 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.331337    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:10.651812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.651736    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"]
Oct 30 21:54:10.656257 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:10.656215    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd707cf_0987_49ae_a504_7dcd1226c050.slice/crio-32cc8ba5e4e8298ae4f507d5ba3aa7553fb3af2881845e7a5375ef8e201b2ce2 WatchSource:0}: Error finding container 32cc8ba5e4e8298ae4f507d5ba3aa7553fb3af2881845e7a5375ef8e201b2ce2: Status 404 returned error can't find the container with id 32cc8ba5e4e8298ae4f507d5ba3aa7553fb3af2881845e7a5375ef8e201b2ce2
Oct 30 21:54:10.859125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.859078    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c" event={"ID":"efd707cf-0987-49ae-a504-7dcd1226c050","Type":"ContainerStarted","Data":"c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688"}
Oct 30 21:54:10.859332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.859147    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c" event={"ID":"efd707cf-0987-49ae-a504-7dcd1226c050","Type":"ContainerStarted","Data":"32cc8ba5e4e8298ae4f507d5ba3aa7553fb3af2881845e7a5375ef8e201b2ce2"}
Oct 30 21:54:10.859332 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:10.859188    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:12.098833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.098777    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c" podStartSLOduration=3.098762527 podStartE2EDuration="3.098762527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.912148598 +0000 UTC m=+2633.834606898" watchObservedRunningTime="2026-10-30 21:54:12.098762527 +0000 UTC m=+2635.021220824"
Oct 30 21:54:12.102554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.102521    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"]
Oct 30 21:54:12.107742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.107718    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.127339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.127309    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"]
Oct 30 21:54:12.133187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.133155    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"routing-overlay\""
Oct 30 21:54:12.133187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.133178    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"payload-processing-external-model-e2e--d53c5c12-dockercfg-fbtpb\""
Oct 30 21:54:12.133187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.133155    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"openshift-service-ca.crt\""
Oct 30 21:54:12.133479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.133210    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"kube-root-ca.crt\""
Oct 30 21:54:12.133479 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.133210    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"e2e-cmp-legacy-a35dae-provider-cred\""
Oct 30 21:54:12.133658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.133630    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-427a6e\""
Oct 30 21:54:12.186430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.186380    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.186664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.186436    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.186664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.186597    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.186664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.186658    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nvqv\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.307889    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.307951    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.308088    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.308128    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nvqv\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.309556    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.315628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.310905    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.340637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.338111    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nvqv\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.361671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.360317    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.427840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.427797    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.749272 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.748913    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"]
Oct 30 21:54:12.751999 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:12.751968    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d600247_d43d_48d4_904b_f79f6cd02872.slice/crio-dffce6108fa3d030b54fed935b949a9ccfca1b67c82dfac96d23b0db78f5c6c5 WatchSource:0}: Error finding container dffce6108fa3d030b54fed935b949a9ccfca1b67c82dfac96d23b0db78f5c6c5: Status 404 returned error can't find the container with id dffce6108fa3d030b54fed935b949a9ccfca1b67c82dfac96d23b0db78f5c6c5
Oct 30 21:54:12.878682 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.878643    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5" event={"ID":"3d600247-d43d-48d4-904b-f79f6cd02872","Type":"ContainerStarted","Data":"bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f"}
Oct 30 21:54:12.878873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.878710    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:12.878873 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.878727    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5" event={"ID":"3d600247-d43d-48d4-904b-f79f6cd02872","Type":"ContainerStarted","Data":"dffce6108fa3d030b54fed935b949a9ccfca1b67c82dfac96d23b0db78f5c6c5"}
Oct 30 21:54:12.921624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:12.921553    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5" podStartSLOduration=0.921539219 podStartE2EDuration="921.539219ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.917287217 +0000 UTC m=+2635.839745515" watchObservedRunningTime="2026-10-30 21:54:12.921539219 +0000 UTC m=+2635.843997517"
Oct 30 21:54:17.703594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.703553    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"]
Oct 30 21:54:17.708573 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.708549    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.729964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.729929    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"]
Oct 30 21:54:17.736468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.736438    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:17.736468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.736461    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:17.736733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.736509    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:17.736733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.736581    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:54:17.743951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.743917    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qq7\" (UniqueName: \"kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.744181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.744154    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.744426 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.744401    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.845953 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.845917    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qq7\" (UniqueName: \"kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.846154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.846038    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.846251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.846155    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.846858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.846836    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.848460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.848427    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:17.865565 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:17.865538    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qq7\" (UniqueName: \"kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7\") pod \"maas-api-revoke-sub-d99fe8a9-v69z9\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:18.024328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:18.024290    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:18.334648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:18.334619    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"]
Oct 30 21:54:18.338015 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:18.337978    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408e97d1_7502_47b6_9c05_8bdb948338e2.slice/crio-0d6071e7599d6bcd1a578724559d89c8ad2f2a15409d78d747f49b4d66452637 WatchSource:0}: Error finding container 0d6071e7599d6bcd1a578724559d89c8ad2f2a15409d78d747f49b4d66452637: Status 404 returned error can't find the container with id 0d6071e7599d6bcd1a578724559d89c8ad2f2a15409d78d747f49b4d66452637
Oct 30 21:54:18.937174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:18.937138    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9" event={"ID":"408e97d1-7502-47b6-9c05-8bdb948338e2","Type":"ContainerStarted","Data":"fb7ec23fb895bd9d7984ea4e48e3692a2e435cfbd64bc2aeda5592fae7cd80f5"}
Oct 30 21:54:18.937642 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:18.937198    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9" event={"ID":"408e97d1-7502-47b6-9c05-8bdb948338e2","Type":"ContainerStarted","Data":"0d6071e7599d6bcd1a578724559d89c8ad2f2a15409d78d747f49b4d66452637"}
Oct 30 21:54:18.993595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:18.993490    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9" podStartSLOduration=1.993476624 podStartE2EDuration="1.993476624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:18.979193467 +0000 UTC m=+2641.901651767" watchObservedRunningTime="2026-10-30 21:54:18.993476624 +0000 UTC m=+2641.915934922"
Oct 30 21:54:19.947229 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:19.947195    6695 generic.go:356] "Generic (PLEG): container finished" podID="408e97d1-7502-47b6-9c05-8bdb948338e2" containerID="fb7ec23fb895bd9d7984ea4e48e3692a2e435cfbd64bc2aeda5592fae7cd80f5" exitCode=0
Oct 30 21:54:19.947658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:19.947270    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9" event={"ID":"408e97d1-7502-47b6-9c05-8bdb948338e2","Type":"ContainerDied","Data":"fb7ec23fb895bd9d7984ea4e48e3692a2e435cfbd64bc2aeda5592fae7cd80f5"}
Oct 30 21:54:21.126031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.126002    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:21.196295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.196259    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7\" (UniqueName: \"kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7\") pod \"408e97d1-7502-47b6-9c05-8bdb948338e2\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") "
Oct 30 21:54:21.196570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.196547    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca\") pod \"408e97d1-7502-47b6-9c05-8bdb948338e2\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") "
Oct 30 21:54:21.196694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.196670    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca\") pod \"408e97d1-7502-47b6-9c05-8bdb948338e2\" (UID: \"408e97d1-7502-47b6-9c05-8bdb948338e2\") "
Oct 30 21:54:21.197023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.196993    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca" pod "408e97d1-7502-47b6-9c05-8bdb948338e2" (UID: "408e97d1-7502-47b6-9c05-8bdb948338e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:21.197386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.197355    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:21.199064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.199034    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7" pod "408e97d1-7502-47b6-9c05-8bdb948338e2" (UID: "408e97d1-7502-47b6-9c05-8bdb948338e2"). InnerVolumeSpecName "kube-api-access-79qq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:21.199064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.199045    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca" pod "408e97d1-7502-47b6-9c05-8bdb948338e2" (UID: "408e97d1-7502-47b6-9c05-8bdb948338e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:21.299277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.299242    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/408e97d1-7502-47b6-9c05-8bdb948338e2-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:21.299277 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.299273    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qq7\" (UniqueName: \"kubernetes.io/projected/408e97d1-7502-47b6-9c05-8bdb948338e2-kube-api-access-79qq7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:21.872169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.872126    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:54:21.963043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.963004    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"]
Oct 30 21:54:21.971615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.971565    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"
Oct 30 21:54:21.971820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.971615    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9" event={"ID":"408e97d1-7502-47b6-9c05-8bdb948338e2","Type":"ContainerDied","Data":"0d6071e7599d6bcd1a578724559d89c8ad2f2a15409d78d747f49b4d66452637"}
Oct 30 21:54:21.971820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:21.971675    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d6071e7599d6bcd1a578724559d89c8ad2f2a15409d78d747f49b4d66452637"
Oct 30 21:54:23.890896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:23.890863    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:54:26.963721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:26.963669    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm" podUID="7e2483d3-3a33-4dcd-b578-838b2887abd4" containerName="payload-processing" containerID="cri-o://e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d" gracePeriod=25
Oct 30 21:54:27.256493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.256463    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:54:27.391311 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.391270    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials\") pod \"7e2483d3-3a33-4dcd-b578-838b2887abd4\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") "
Oct 30 21:54:27.391526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.391402    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume\") pod \"7e2483d3-3a33-4dcd-b578-838b2887abd4\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") "
Oct 30 21:54:27.391526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.391473    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume\") pod \"7e2483d3-3a33-4dcd-b578-838b2887abd4\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") "
Oct 30 21:54:27.391526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.391519    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl\") pod \"7e2483d3-3a33-4dcd-b578-838b2887abd4\" (UID: \"7e2483d3-3a33-4dcd-b578-838b2887abd4\") "
Oct 30 21:54:27.391864 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.391837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume" pod "7e2483d3-3a33-4dcd-b578-838b2887abd4" (UID: "7e2483d3-3a33-4dcd-b578-838b2887abd4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:27.391929 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.391867    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume" pod "7e2483d3-3a33-4dcd-b578-838b2887abd4" (UID: "7e2483d3-3a33-4dcd-b578-838b2887abd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:27.392316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.392290    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:27.392316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.392312    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e2483d3-3a33-4dcd-b578-838b2887abd4-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:27.393550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.393524    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials" pod "7e2483d3-3a33-4dcd-b578-838b2887abd4" (UID: "7e2483d3-3a33-4dcd-b578-838b2887abd4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:27.393732 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.393714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl" pod "7e2483d3-3a33-4dcd-b578-838b2887abd4" (UID: "7e2483d3-3a33-4dcd-b578-838b2887abd4"). InnerVolumeSpecName "kube-api-access-nfrnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:27.493649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.493537    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:27.493649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:27.493576    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrnl\" (UniqueName: \"kubernetes.io/projected/7e2483d3-3a33-4dcd-b578-838b2887abd4-kube-api-access-nfrnl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:28.034821 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.034785    6695 generic.go:356] "Generic (PLEG): container finished" podID="7e2483d3-3a33-4dcd-b578-838b2887abd4" containerID="e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d" exitCode=0
Oct 30 21:54:28.035288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.034837    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"
Oct 30 21:54:28.035288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.034868    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm" event={"ID":"7e2483d3-3a33-4dcd-b578-838b2887abd4","Type":"ContainerDied","Data":"e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d"}
Oct 30 21:54:28.035288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.034923    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm" event={"ID":"7e2483d3-3a33-4dcd-b578-838b2887abd4","Type":"ContainerDied","Data":"a17fa6276a7626b15f6f3b1e2928a25c9a9ef49801e99cf4dbe51660e0f387f7"}
Oct 30 21:54:28.035288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.034945    6695 scope.go:122] "RemoveContainer" containerID="e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d"
Oct 30 21:54:28.046839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.046815    6695 scope.go:122] "RemoveContainer" containerID="e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d"
Oct 30 21:54:28.047205 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:54:28.047183    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d\": container with ID starting with e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d not found: ID does not exist" containerID="e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d"
Oct 30 21:54:28.047317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.047216    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d"} err="failed to get container status \"e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d\": rpc error: code = NotFound desc = could not find container \"e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d\": container with ID starting with e26152ac31b5916d633ffd68764040c917fcabaa4e035ef67d3b787141c27d0d not found: ID does not exist"
Oct 30 21:54:28.085216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.085176    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"]
Oct 30 21:54:28.096284 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:28.096252    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-8576r74wm"]
Oct 30 21:54:29.205443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.205390    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"]
Oct 30 21:54:29.206823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.206788    6695 cpu_manager.go:414] "containerMap: removing container" podUID="408e97d1-7502-47b6-9c05-8bdb948338e2" containerName="revoke-keys"
Oct 30 21:54:29.206823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.206813    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408e97d1-7502-47b6-9c05-8bdb948338e2" containerName="revoke-keys"
Oct 30 21:54:29.207021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.206933    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7e2483d3-3a33-4dcd-b578-838b2887abd4" containerName="payload-processing"
Oct 30 21:54:29.207021 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.206952    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2483d3-3a33-4dcd-b578-838b2887abd4" containerName="payload-processing"
Oct 30 21:54:29.207186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.207167    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="408e97d1-7502-47b6-9c05-8bdb948338e2" containerName="revoke-keys"
Oct 30 21:54:29.207274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.207208    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2483d3-3a33-4dcd-b578-838b2887abd4" containerName="payload-processing"
Oct 30 21:54:29.213724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.213696    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.215715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.215689    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"]
Oct 30 21:54:29.227978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.227944    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2483d3-3a33-4dcd-b578-838b2887abd4" path="/var/lib/kubelet/pods/7e2483d3-3a33-4dcd-b578-838b2887abd4/volumes"
Oct 30 21:54:29.239690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.239662    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:29.239870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.239760    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:54:29.239870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.239787    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:29.239870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.239826    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:29.317068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.317029    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.317287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.317237    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.317431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.317412    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjrj\" (UniqueName: \"kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.419403 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.419365    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjrj\" (UniqueName: \"kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.419637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.419477    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.419637 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.419567    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.420375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.420345    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.421964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.421946    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.438638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.438590    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjrj\" (UniqueName: \"kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj\") pod \"maas-api-revoke-sub-0254599e-j56ht\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.535275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.535243    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:29.846358 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:29.846237    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"]
Oct 30 21:54:29.851276 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:29.851247    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41e3d3f_cbe0_49df_9794_a7db9bffd145.slice/crio-ee352c0ea5a5cc048aa927ed659222805f162fee4b452d0371d2428f20a46e8e WatchSource:0}: Error finding container ee352c0ea5a5cc048aa927ed659222805f162fee4b452d0371d2428f20a46e8e: Status 404 returned error can't find the container with id ee352c0ea5a5cc048aa927ed659222805f162fee4b452d0371d2428f20a46e8e
Oct 30 21:54:30.052940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:30.052906    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht" event={"ID":"a41e3d3f-cbe0-49df-9794-a7db9bffd145","Type":"ContainerStarted","Data":"ee352c0ea5a5cc048aa927ed659222805f162fee4b452d0371d2428f20a46e8e"}
Oct 30 21:54:31.064499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:31.064450    6695 generic.go:356] "Generic (PLEG): container finished" podID="a41e3d3f-cbe0-49df-9794-a7db9bffd145" containerID="f01f465ac812ca0f280a8d399e2854f6166a785c720c1d34509e5704eb802da0" exitCode=0
Oct 30 21:54:31.064944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:31.064512    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht" event={"ID":"a41e3d3f-cbe0-49df-9794-a7db9bffd145","Type":"ContainerDied","Data":"f01f465ac812ca0f280a8d399e2854f6166a785c720c1d34509e5704eb802da0"}
Oct 30 21:54:31.077125 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:54:31.077041    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:54:32.249885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.249849    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:32.271380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.271344    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca\") pod \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") "
Oct 30 21:54:32.271576 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.271544    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj\" (UniqueName: \"kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj\") pod \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") "
Oct 30 21:54:32.271710 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.271693    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca\") pod \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\" (UID: \"a41e3d3f-cbe0-49df-9794-a7db9bffd145\") "
Oct 30 21:54:32.271779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.271707    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca" pod "a41e3d3f-cbe0-49df-9794-a7db9bffd145" (UID: "a41e3d3f-cbe0-49df-9794-a7db9bffd145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:32.272298 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.272260    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:32.273794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.273770    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca" pod "a41e3d3f-cbe0-49df-9794-a7db9bffd145" (UID: "a41e3d3f-cbe0-49df-9794-a7db9bffd145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:32.273922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.273903    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj" pod "a41e3d3f-cbe0-49df-9794-a7db9bffd145" (UID: "a41e3d3f-cbe0-49df-9794-a7db9bffd145"). InnerVolumeSpecName "kube-api-access-lvjrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:32.373794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.373706    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjrj\" (UniqueName: \"kubernetes.io/projected/a41e3d3f-cbe0-49df-9794-a7db9bffd145-kube-api-access-lvjrj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:32.373794 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:32.373736    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41e3d3f-cbe0-49df-9794-a7db9bffd145-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:33.081068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:33.081037    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"
Oct 30 21:54:33.081068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:33.081043    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht" event={"ID":"a41e3d3f-cbe0-49df-9794-a7db9bffd145","Type":"ContainerDied","Data":"ee352c0ea5a5cc048aa927ed659222805f162fee4b452d0371d2428f20a46e8e"}
Oct 30 21:54:33.081290 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:33.081097    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee352c0ea5a5cc048aa927ed659222805f162fee4b452d0371d2428f20a46e8e"
Oct 30 21:54:40.818863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.818818    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"]
Oct 30 21:54:40.820016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.819995    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a41e3d3f-cbe0-49df-9794-a7db9bffd145" containerName="revoke-keys"
Oct 30 21:54:40.820016 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.820016    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41e3d3f-cbe0-49df-9794-a7db9bffd145" containerName="revoke-keys"
Oct 30 21:54:40.820137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.820125    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41e3d3f-cbe0-49df-9794-a7db9bffd145" containerName="revoke-keys"
Oct 30 21:54:40.824280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.824248    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:40.857110 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.857067    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"]
Oct 30 21:54:40.921428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.921389    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:40.921706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.921446    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:40.921706 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.921586    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnlfr\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:40.921844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.921770    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:40.982436 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.982393    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"]
Oct 30 21:54:40.988798 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:40.988768    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.003944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.003902    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:41.005018 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.004981    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:41.005329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.005313    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:54:41.015641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.012640    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"]
Oct 30 21:54:41.016160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.016131    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:41.023145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.023116    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.023313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.023251    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.023391 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.023309    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.023466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.023410    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnlfr\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.024107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.024078    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.024504 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.024473    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.028636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.028614    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.044222 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.044184    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnlfr\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.184679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.159493    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvt7\" (UniqueName: \"kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.184679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.159550    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.184679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.159820    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:41.184679 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.177835    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.279511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.279478    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.280483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.280454    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvt7\" (UniqueName: \"kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.280698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.280682    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.281870 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.281831    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.284371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.284320    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.304434 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.304397    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvt7\" (UniqueName: \"kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7\") pod \"maas-api-revoke-sub-9e13ac79-tg9bs\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.321218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.321183    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:41.348788 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.347899    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"]
Oct 30 21:54:41.354429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.354399    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.389500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.384402    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.389500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.384664    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.389500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.384739    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmk5\" (UniqueName: \"kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.389500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.384935    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"]
Oct 30 21:54:41.485986 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.485897    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.486177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.485986    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmk5\" (UniqueName: \"kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.486177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.486118    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.486973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.486942    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.488526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.488506    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.507701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.507676    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmk5\" (UniqueName: \"kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5\") pod \"maas-api-revoke-sub-744700f5-7sdzk\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.606341 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.606313    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"]
Oct 30 21:54:41.608451 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:41.608419    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0577fc_2bfe_4c97_a08e_5f7b12ce32f8.slice/crio-5392922110b7be9f6700f8a7e583fa6bf26c275ee21b6f067c1ef7b0117596d8 WatchSource:0}: Error finding container 5392922110b7be9f6700f8a7e583fa6bf26c275ee21b6f067c1ef7b0117596d8: Status 404 returned error can't find the container with id 5392922110b7be9f6700f8a7e583fa6bf26c275ee21b6f067c1ef7b0117596d8
Oct 30 21:54:41.673677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.672837    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"]
Oct 30 21:54:41.678410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:41.678389    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:41.678569 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:41.678540    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe863741_1fc7_4af3_b7d4_a288f0f82731.slice/crio-c74c28ee34d5c35b9e8bc8490267008f3bfa714254c07265b9dc5ec8fe5af289 WatchSource:0}: Error finding container c74c28ee34d5c35b9e8bc8490267008f3bfa714254c07265b9dc5ec8fe5af289: Status 404 returned error can't find the container with id c74c28ee34d5c35b9e8bc8490267008f3bfa714254c07265b9dc5ec8fe5af289
Oct 30 21:54:42.063977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.063935    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"]
Oct 30 21:54:42.228006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.227699    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6" event={"ID":"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8","Type":"ContainerStarted","Data":"f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393"}
Oct 30 21:54:42.228006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.227763    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6" event={"ID":"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8","Type":"ContainerStarted","Data":"5392922110b7be9f6700f8a7e583fa6bf26c275ee21b6f067c1ef7b0117596d8"}
Oct 30 21:54:42.228006 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.227825    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:42.229579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.229551    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs" event={"ID":"be863741-1fc7-4af3-b7d4-a288f0f82731","Type":"ContainerStarted","Data":"c74c28ee34d5c35b9e8bc8490267008f3bfa714254c07265b9dc5ec8fe5af289"}
Oct 30 21:54:42.231118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.231092    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk" event={"ID":"27bb615f-cf21-4fa6-bc03-d7d96f422447","Type":"ContainerStarted","Data":"198208403dd3849a076962f00f37e1b6dd5e462ce63a8756c66cd4b96962af95"}
Oct 30 21:54:42.268383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:42.268303    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6" podStartSLOduration=2.268283828 podStartE2EDuration="2.268283828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:42.265703978 +0000 UTC m=+2665.188162276" watchObservedRunningTime="2026-10-30 21:54:42.268283828 +0000 UTC m=+2665.190742126"
Oct 30 21:54:43.241248 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.241162    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk" event={"ID":"27bb615f-cf21-4fa6-bc03-d7d96f422447","Type":"ContainerStarted","Data":"b0343ae6b9e8f6b43e4840cf8d0e9a32d69244d271173256aec3858a1a4c4062"}
Oct 30 21:54:43.243034 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.243006    6695 generic.go:356] "Generic (PLEG): container finished" podID="be863741-1fc7-4af3-b7d4-a288f0f82731" containerID="99da5c6a31846b568b99638794a96df9c0d5b907222e96a34b69ece37259ae3c" exitCode=0
Oct 30 21:54:43.243145 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.243072    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs" event={"ID":"be863741-1fc7-4af3-b7d4-a288f0f82731","Type":"ContainerDied","Data":"99da5c6a31846b568b99638794a96df9c0d5b907222e96a34b69ece37259ae3c"}
Oct 30 21:54:43.281626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.279998    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk" podStartSLOduration=2.279940286 podStartE2EDuration="2.279940286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:43.277571868 +0000 UTC m=+2666.200030168" watchObservedRunningTime="2026-10-30 21:54:43.279940286 +0000 UTC m=+2666.202398585"
Oct 30 21:54:43.843102 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.843060    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"]
Oct 30 21:54:43.847958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.847936    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:43.862160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.862094    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"agc-cred-68bf0c456c5f\""
Oct 30 21:54:43.866139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.866114    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"]
Oct 30 21:54:43.983586 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.983545    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8l9h\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:43.983779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.983644    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:43.983844 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.983794    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:43.983951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:43.983930    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.085648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.085577    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8l9h\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.085828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.085673    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.085828 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.085781    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.085967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.085879    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.086582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.086548    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.086737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.086706    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.088708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.088682    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.104525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.104461    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8l9h\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.165570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.165529    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:44.260806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.260770    6695 generic.go:356] "Generic (PLEG): container finished" podID="27bb615f-cf21-4fa6-bc03-d7d96f422447" containerID="b0343ae6b9e8f6b43e4840cf8d0e9a32d69244d271173256aec3858a1a4c4062" exitCode=0
Oct 30 21:54:44.262029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.261997    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk" event={"ID":"27bb615f-cf21-4fa6-bc03-d7d96f422447","Type":"ContainerDied","Data":"b0343ae6b9e8f6b43e4840cf8d0e9a32d69244d271173256aec3858a1a4c4062"}
Oct 30 21:54:44.473154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.473125    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:44.525498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.525143    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"]
Oct 30 21:54:44.595044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.595013    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7\" (UniqueName: \"kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7\") pod \"be863741-1fc7-4af3-b7d4-a288f0f82731\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") "
Oct 30 21:54:44.595242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.595061    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca\") pod \"be863741-1fc7-4af3-b7d4-a288f0f82731\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") "
Oct 30 21:54:44.595242 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.595134    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca\") pod \"be863741-1fc7-4af3-b7d4-a288f0f82731\" (UID: \"be863741-1fc7-4af3-b7d4-a288f0f82731\") "
Oct 30 21:54:44.595571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.595542    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca" pod "be863741-1fc7-4af3-b7d4-a288f0f82731" (UID: "be863741-1fc7-4af3-b7d4-a288f0f82731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:44.595895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.595876    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:44.597777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.597754    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7" pod "be863741-1fc7-4af3-b7d4-a288f0f82731" (UID: "be863741-1fc7-4af3-b7d4-a288f0f82731"). InnerVolumeSpecName "kube-api-access-zwvt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:44.597777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.597755    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca" pod "be863741-1fc7-4af3-b7d4-a288f0f82731" (UID: "be863741-1fc7-4af3-b7d4-a288f0f82731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:44.697812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.697732    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvt7\" (UniqueName: \"kubernetes.io/projected/be863741-1fc7-4af3-b7d4-a288f0f82731-kube-api-access-zwvt7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:44.697812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:44.697760    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be863741-1fc7-4af3-b7d4-a288f0f82731-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:45.271198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.271159    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8" event={"ID":"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26","Type":"ContainerStarted","Data":"c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26"}
Oct 30 21:54:45.271715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.271223    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8" event={"ID":"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26","Type":"ContainerStarted","Data":"a9fe454772f13fcce8be001b60c19780fc831e35faf10ea90904460783060bad"}
Oct 30 21:54:45.271715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.271270    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:45.272772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.272747    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"
Oct 30 21:54:45.272910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.272773    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs" event={"ID":"be863741-1fc7-4af3-b7d4-a288f0f82731","Type":"ContainerDied","Data":"c74c28ee34d5c35b9e8bc8490267008f3bfa714254c07265b9dc5ec8fe5af289"}
Oct 30 21:54:45.272910 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.272811    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74c28ee34d5c35b9e8bc8490267008f3bfa714254c07265b9dc5ec8fe5af289"
Oct 30 21:54:45.315481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.315431    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8" podStartSLOduration=2.315414945 podStartE2EDuration="2.315414945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:45.310299712 +0000 UTC m=+2668.232758022" watchObservedRunningTime="2026-10-30 21:54:45.315414945 +0000 UTC m=+2668.237873243"
Oct 30 21:54:45.429517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.429492    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:45.509421 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.509305    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca\") pod \"27bb615f-cf21-4fa6-bc03-d7d96f422447\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") "
Oct 30 21:54:45.509595 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.509472    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5\" (UniqueName: \"kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5\") pod \"27bb615f-cf21-4fa6-bc03-d7d96f422447\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") "
Oct 30 21:54:45.509688 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.509664    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca\") pod \"27bb615f-cf21-4fa6-bc03-d7d96f422447\" (UID: \"27bb615f-cf21-4fa6-bc03-d7d96f422447\") "
Oct 30 21:54:45.509813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.509781    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca" pod "27bb615f-cf21-4fa6-bc03-d7d96f422447" (UID: "27bb615f-cf21-4fa6-bc03-d7d96f422447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:45.510428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.510406    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:45.512975 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.512943    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca" pod "27bb615f-cf21-4fa6-bc03-d7d96f422447" (UID: "27bb615f-cf21-4fa6-bc03-d7d96f422447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:45.513261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.513235    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5" pod "27bb615f-cf21-4fa6-bc03-d7d96f422447" (UID: "27bb615f-cf21-4fa6-bc03-d7d96f422447"). InnerVolumeSpecName "kube-api-access-9wmk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:45.611902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.611869    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wmk5\" (UniqueName: \"kubernetes.io/projected/27bb615f-cf21-4fa6-bc03-d7d96f422447-kube-api-access-9wmk5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:45.611902 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:45.611896    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27bb615f-cf21-4fa6-bc03-d7d96f422447-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:46.283156 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:46.283124    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"
Oct 30 21:54:46.283557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:46.283160    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk" event={"ID":"27bb615f-cf21-4fa6-bc03-d7d96f422447","Type":"ContainerDied","Data":"198208403dd3849a076962f00f37e1b6dd5e462ce63a8756c66cd4b96962af95"}
Oct 30 21:54:46.283557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:46.283207    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="198208403dd3849a076962f00f37e1b6dd5e462ce63a8756c66cd4b96962af95"
Oct 30 21:54:52.507627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.504485    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"]
Oct 30 21:54:52.507627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.507474    6695 cpu_manager.go:414] "containerMap: removing container" podUID="be863741-1fc7-4af3-b7d4-a288f0f82731" containerName="revoke-keys"
Oct 30 21:54:52.507627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.507501    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be863741-1fc7-4af3-b7d4-a288f0f82731" containerName="revoke-keys"
Oct 30 21:54:52.508280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.507767    6695 cpu_manager.go:414] "containerMap: removing container" podUID="27bb615f-cf21-4fa6-bc03-d7d96f422447" containerName="revoke-keys"
Oct 30 21:54:52.508280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.507784    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27bb615f-cf21-4fa6-bc03-d7d96f422447" containerName="revoke-keys"
Oct 30 21:54:52.508280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.508005    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="27bb615f-cf21-4fa6-bc03-d7d96f422447" containerName="revoke-keys"
Oct 30 21:54:52.508280 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.508028    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="be863741-1fc7-4af3-b7d4-a288f0f82731" containerName="revoke-keys"
Oct 30 21:54:52.521630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.515262    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.560638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.560516    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:54:52.562186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.562094    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.562186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.562141    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsh2\" (UniqueName: \"kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.562396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.562233    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.590625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.580445    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:54:52.590625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.581051    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:52.590625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.581170    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:54:52.593367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.590977    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"]
Oct 30 21:54:52.663653 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.663620    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.663877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.663656    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsh2\" (UniqueName: \"kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.663877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.663687    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.664513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.664486    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.666154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.666133    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.683043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.683015    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsh2\" (UniqueName: \"kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2\") pod \"maas-api-revoke-sub-cb9fabe4-xx2wm\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.852237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.852197    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"]
Oct 30 21:54:52.855747 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.855717    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:52.881109 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.881069    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"]
Oct 30 21:54:52.884342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.884314    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:52.886804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.886779    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:52.909793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.909757    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"]
Oct 30 21:54:52.956961 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.956923    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"]
Oct 30 21:54:52.962957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:52.962913    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.058626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.050058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.058626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.050122    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2xs\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.058626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.050288    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.058626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.050340    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.058626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.051081    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"]
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152138    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152212    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2xs\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152254    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152293    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152333    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprgl\" (UniqueName: \"kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152456    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.152819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.152497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.153500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.153469    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.160296 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.155017    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.164466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.164432    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.174295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.174261    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2xs\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.322667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.253882    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:53.322667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.259794    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.322667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.259847    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.322667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.259882    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprgl\" (UniqueName: \"kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.322667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.319363    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.336636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.336562    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"]
Oct 30 21:54:53.356628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.353879    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.356628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.353959    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprgl\" (UniqueName: \"kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl\") pod \"maas-api-revoke-sub-0d47766f-kwfms\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.390338 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:53.390295    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1763a0_fc71_4e23_889b_0649db2465ea.slice/crio-c84ce38e4d6373f5639284c17a5d6956edd7d1116c800e52a404890b80dffd84 WatchSource:0}: Error finding container c84ce38e4d6373f5639284c17a5d6956edd7d1116c800e52a404890b80dffd84: Status 404 returned error can't find the container with id c84ce38e4d6373f5639284c17a5d6956edd7d1116c800e52a404890b80dffd84
Oct 30 21:54:53.441125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.441093    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"]
Oct 30 21:54:53.451387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.451159    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:53.695514 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.695484    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"]
Oct 30 21:54:53.696945 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:53.696917    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd981eb_4e34_4364_9dab_b6e578a51ef1.slice/crio-eed576bc2fd51a5f671b6e5e51b84ead5265e95a037b05d6f437ea26ad0d682f WatchSource:0}: Error finding container eed576bc2fd51a5f671b6e5e51b84ead5265e95a037b05d6f437ea26ad0d682f: Status 404 returned error can't find the container with id eed576bc2fd51a5f671b6e5e51b84ead5265e95a037b05d6f437ea26ad0d682f
Oct 30 21:54:53.795722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:53.795693    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"]
Oct 30 21:54:53.798097 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:54:53.798065    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa21fc2_dcc1_4a5f_a628_e630065b872f.slice/crio-e3a6c6f647b792ae63a3469a77c8c2ff1601c30c5367a9b7630c047596818ae6 WatchSource:0}: Error finding container e3a6c6f647b792ae63a3469a77c8c2ff1601c30c5367a9b7630c047596818ae6: Status 404 returned error can't find the container with id e3a6c6f647b792ae63a3469a77c8c2ff1601c30c5367a9b7630c047596818ae6
Oct 30 21:54:54.435695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.434750    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx" event={"ID":"8dd981eb-4e34-4364-9dab-b6e578a51ef1","Type":"ContainerStarted","Data":"f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06"}
Oct 30 21:54:54.435695 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.434881    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx" event={"ID":"8dd981eb-4e34-4364-9dab-b6e578a51ef1","Type":"ContainerStarted","Data":"eed576bc2fd51a5f671b6e5e51b84ead5265e95a037b05d6f437ea26ad0d682f"}
Oct 30 21:54:54.439626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.438584    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:54:54.455627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.442653    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms" event={"ID":"bfa21fc2-dcc1-4a5f-a628-e630065b872f","Type":"ContainerStarted","Data":"a1e270e80d8d82c33a958f41bfe9ec95f059767d364b52977633a662c892b6ca"}
Oct 30 21:54:54.455627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.442708    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms" event={"ID":"bfa21fc2-dcc1-4a5f-a628-e630065b872f","Type":"ContainerStarted","Data":"e3a6c6f647b792ae63a3469a77c8c2ff1601c30c5367a9b7630c047596818ae6"}
Oct 30 21:54:54.455627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.445309    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm" event={"ID":"7c1763a0-fc71-4e23-889b-0649db2465ea","Type":"ContainerStarted","Data":"09bdf8be777ee8db57b14b97beef2b519910143baca11ade9a3e5bc5299fa345"}
Oct 30 21:54:54.455627 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.445350    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm" event={"ID":"7c1763a0-fc71-4e23-889b-0649db2465ea","Type":"ContainerStarted","Data":"c84ce38e4d6373f5639284c17a5d6956edd7d1116c800e52a404890b80dffd84"}
Oct 30 21:54:54.489667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.489588    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx" podStartSLOduration=2.489566365 podStartE2EDuration="2.489566365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:54.485050238 +0000 UTC m=+2677.407508528" watchObservedRunningTime="2026-10-30 21:54:54.489566365 +0000 UTC m=+2677.412024664"
Oct 30 21:54:54.537090 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.537022    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms" podStartSLOduration=2.537003658 podStartE2EDuration="2.537003658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:54.532355924 +0000 UTC m=+2677.454814222" watchObservedRunningTime="2026-10-30 21:54:54.537003658 +0000 UTC m=+2677.459461958"
Oct 30 21:54:54.583719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:54.583668    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm" podStartSLOduration=2.583653024 podStartE2EDuration="2.583653024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:54.58028911 +0000 UTC m=+2677.502747408" watchObservedRunningTime="2026-10-30 21:54:54.583653024 +0000 UTC m=+2677.506111321"
Oct 30 21:54:55.454620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:55.454573    6695 generic.go:356] "Generic (PLEG): container finished" podID="bfa21fc2-dcc1-4a5f-a628-e630065b872f" containerID="a1e270e80d8d82c33a958f41bfe9ec95f059767d364b52977633a662c892b6ca" exitCode=0
Oct 30 21:54:55.455108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:55.454670    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms" event={"ID":"bfa21fc2-dcc1-4a5f-a628-e630065b872f","Type":"ContainerDied","Data":"a1e270e80d8d82c33a958f41bfe9ec95f059767d364b52977633a662c892b6ca"}
Oct 30 21:54:55.456407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:55.456384    6695 generic.go:356] "Generic (PLEG): container finished" podID="7c1763a0-fc71-4e23-889b-0649db2465ea" containerID="09bdf8be777ee8db57b14b97beef2b519910143baca11ade9a3e5bc5299fa345" exitCode=0
Oct 30 21:54:55.456493 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:55.456469    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm" event={"ID":"7c1763a0-fc71-4e23-889b-0649db2465ea","Type":"ContainerDied","Data":"09bdf8be777ee8db57b14b97beef2b519910143baca11ade9a3e5bc5299fa345"}
Oct 30 21:54:56.320118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:56.318854    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:54:56.411755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:56.410530    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"]
Oct 30 21:54:56.838674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:56.826128    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:56.900541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:56.900417    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:57.006640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.004919    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2\" (UniqueName: \"kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2\") pod \"7c1763a0-fc71-4e23-889b-0649db2465ea\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") "
Oct 30 21:54:57.006640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.005072    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca\") pod \"7c1763a0-fc71-4e23-889b-0649db2465ea\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") "
Oct 30 21:54:57.020648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.020510    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca\") pod \"7c1763a0-fc71-4e23-889b-0649db2465ea\" (UID: \"7c1763a0-fc71-4e23-889b-0649db2465ea\") "
Oct 30 21:54:57.027499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.027181    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca" pod "7c1763a0-fc71-4e23-889b-0649db2465ea" (UID: "7c1763a0-fc71-4e23-889b-0649db2465ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:57.028473 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.028418    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl\" (UniqueName: \"kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl\") pod \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") "
Oct 30 21:54:57.030240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.030205    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:57.038634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.038580    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl" pod "bfa21fc2-dcc1-4a5f-a628-e630065b872f" (UID: "bfa21fc2-dcc1-4a5f-a628-e630065b872f"). InnerVolumeSpecName "kube-api-access-qprgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:57.041699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.041668    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca" pod "7c1763a0-fc71-4e23-889b-0649db2465ea" (UID: "7c1763a0-fc71-4e23-889b-0649db2465ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:57.084661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.079301    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2" pod "7c1763a0-fc71-4e23-889b-0649db2465ea" (UID: "7c1763a0-fc71-4e23-889b-0649db2465ea"). InnerVolumeSpecName "kube-api-access-8lsh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:57.132125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.131146    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca\") pod \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") "
Oct 30 21:54:57.132125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.131239    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca\") pod \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\" (UID: \"bfa21fc2-dcc1-4a5f-a628-e630065b872f\") "
Oct 30 21:54:57.132125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.132056    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsh2\" (UniqueName: \"kubernetes.io/projected/7c1763a0-fc71-4e23-889b-0649db2465ea-kube-api-access-8lsh2\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:57.132125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.132080    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1763a0-fc71-4e23-889b-0649db2465ea-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:57.132125 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.132094    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprgl\" (UniqueName: \"kubernetes.io/projected/bfa21fc2-dcc1-4a5f-a628-e630065b872f-kube-api-access-qprgl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:57.132506 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.132309    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca" pod "bfa21fc2-dcc1-4a5f-a628-e630065b872f" (UID: "bfa21fc2-dcc1-4a5f-a628-e630065b872f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:57.138486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.138440    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca" pod "bfa21fc2-dcc1-4a5f-a628-e630065b872f" (UID: "bfa21fc2-dcc1-4a5f-a628-e630065b872f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:57.283917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.283878    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:57.288685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.288654    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfa21fc2-dcc1-4a5f-a628-e630065b872f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:57.503028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.502981    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm" event={"ID":"7c1763a0-fc71-4e23-889b-0649db2465ea","Type":"ContainerDied","Data":"c84ce38e4d6373f5639284c17a5d6956edd7d1116c800e52a404890b80dffd84"}
Oct 30 21:54:57.503234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.503059    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84ce38e4d6373f5639284c17a5d6956edd7d1116c800e52a404890b80dffd84"
Oct 30 21:54:57.503234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.503142    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"
Oct 30 21:54:57.512093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.511090    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms" event={"ID":"bfa21fc2-dcc1-4a5f-a628-e630065b872f","Type":"ContainerDied","Data":"e3a6c6f647b792ae63a3469a77c8c2ff1601c30c5367a9b7630c047596818ae6"}
Oct 30 21:54:57.512093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.511144    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a6c6f647b792ae63a3469a77c8c2ff1601c30c5367a9b7630c047596818ae6"
Oct 30 21:54:57.512093 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.511203    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"
Oct 30 21:54:57.853035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:57.852995    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6" podUID="0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" containerName="payload-processing" containerID="cri-o://f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393" gracePeriod=25
Oct 30 21:54:58.137218 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.137184    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:58.215405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.215365    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume\") pod \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") "
Oct 30 21:54:58.215592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.215417    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials\") pod \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") "
Oct 30 21:54:58.215592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.215444    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr\") pod \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") "
Oct 30 21:54:58.215592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.215563    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume\") pod \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\" (UID: \"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8\") "
Oct 30 21:54:58.216320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.216291    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume" pod "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" (UID: "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.216661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.216634    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume" pod "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" (UID: "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.216773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.216742    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.218266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.218229    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr" pod "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" (UID: "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8"). InnerVolumeSpecName "kube-api-access-rnlfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.218393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.218284    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials" pod "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" (UID: "0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.318259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.318221    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.318259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.318253    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnlfr\" (UniqueName: \"kubernetes.io/projected/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-kube-api-access-rnlfr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.318259 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.318262    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.442180 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.442132    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq" podUID="15548e1b-fc98-4ba4-8a15-d2cc85159602" containerName="payload-processing" containerID="cri-o://9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8" gracePeriod=25
Oct 30 21:54:58.537487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.537448    6695 generic.go:356] "Generic (PLEG): container finished" podID="0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" containerID="f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393" exitCode=0
Oct 30 21:54:58.537734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.537505    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"
Oct 30 21:54:58.537734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.537539    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6" event={"ID":"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8","Type":"ContainerDied","Data":"f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393"}
Oct 30 21:54:58.537734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.537625    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6" event={"ID":"0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8","Type":"ContainerDied","Data":"5392922110b7be9f6700f8a7e583fa6bf26c275ee21b6f067c1ef7b0117596d8"}
Oct 30 21:54:58.537734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.537662    6695 scope.go:122] "RemoveContainer" containerID="f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393"
Oct 30 21:54:58.556577 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.556551    6695 scope.go:122] "RemoveContainer" containerID="f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393"
Oct 30 21:54:58.556936 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:54:58.556909    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393\": container with ID starting with f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393 not found: ID does not exist" containerID="f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393"
Oct 30 21:54:58.557027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.556943    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393"} err="failed to get container status \"f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393\": rpc error: code = NotFound desc = could not find container \"f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393\": container with ID starting with f4a2826423485c315fad8a463ad9c81679050f6bc8d04015877a19bc17490393 not found: ID does not exist"
Oct 30 21:54:58.592026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.591992    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"]
Oct 30 21:54:58.599991 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.599956    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-f669h2cv6"]
Oct 30 21:54:58.740807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.740772    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:54:58.825135 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.825043    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr\") pod \"15548e1b-fc98-4ba4-8a15-d2cc85159602\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") "
Oct 30 21:54:58.825291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.825155    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume\") pod \"15548e1b-fc98-4ba4-8a15-d2cc85159602\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") "
Oct 30 21:54:58.825291 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.825260    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials\") pod \"15548e1b-fc98-4ba4-8a15-d2cc85159602\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") "
Oct 30 21:54:58.825377 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.825364    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume\") pod \"15548e1b-fc98-4ba4-8a15-d2cc85159602\" (UID: \"15548e1b-fc98-4ba4-8a15-d2cc85159602\") "
Oct 30 21:54:58.825823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.825795    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume" pod "15548e1b-fc98-4ba4-8a15-d2cc85159602" (UID: "15548e1b-fc98-4ba4-8a15-d2cc85159602"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.826266 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.826244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume" pod "15548e1b-fc98-4ba4-8a15-d2cc85159602" (UID: "15548e1b-fc98-4ba4-8a15-d2cc85159602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.828720 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.828682    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr" pod "15548e1b-fc98-4ba4-8a15-d2cc85159602" (UID: "15548e1b-fc98-4ba4-8a15-d2cc85159602"). InnerVolumeSpecName "kube-api-access-zz2mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.828956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.828926    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials" pod "15548e1b-fc98-4ba4-8a15-d2cc85159602" (UID: "15548e1b-fc98-4ba4-8a15-d2cc85159602"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.932225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.932182    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.932225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.932228    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2mr\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-kube-api-access-zz2mr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.932779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.932246    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15548e1b-fc98-4ba4-8a15-d2cc85159602-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:58.932779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:58.932269    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15548e1b-fc98-4ba4-8a15-d2cc85159602-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:54:59.215888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.215811    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" path="/var/lib/kubelet/pods/0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8/volumes"
Oct 30 21:54:59.547951 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.547912    6695 generic.go:356] "Generic (PLEG): container finished" podID="15548e1b-fc98-4ba4-8a15-d2cc85159602" containerID="9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8" exitCode=0
Oct 30 21:54:59.548111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.547990    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"
Oct 30 21:54:59.548111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.547997    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq" event={"ID":"15548e1b-fc98-4ba4-8a15-d2cc85159602","Type":"ContainerDied","Data":"9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8"}
Oct 30 21:54:59.548111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.548060    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq" event={"ID":"15548e1b-fc98-4ba4-8a15-d2cc85159602","Type":"ContainerDied","Data":"a1eb05ede8c326d74b5f48527e7168606047eb41b8b0eda23e7f02a6e432c39f"}
Oct 30 21:54:59.548111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.548082    6695 scope.go:122] "RemoveContainer" containerID="9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8"
Oct 30 21:54:59.559454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.559435    6695 scope.go:122] "RemoveContainer" containerID="9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8"
Oct 30 21:54:59.559764 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:54:59.559746    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8\": container with ID starting with 9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8 not found: ID does not exist" containerID="9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8"
Oct 30 21:54:59.559852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.559772    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8"} err="failed to get container status \"9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8\": rpc error: code = NotFound desc = could not find container \"9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8\": container with ID starting with 9d5a3eab77fc7b73d050b007d12c9405ff706c22e9163786f9624ae0227b92e8 not found: ID does not exist"
Oct 30 21:54:59.599073 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.599041    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"]
Oct 30 21:54:59.609950 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:54:59.609915    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-74bf776kq"]
Oct 30 21:55:01.075372 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:01.075332    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:55:01.216468 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.216434    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15548e1b-fc98-4ba4-8a15-d2cc85159602" path="/var/lib/kubelet/pods/15548e1b-fc98-4ba4-8a15-d2cc85159602/volumes"
Oct 30 21:55:01.411816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.411711    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs" podUID="3aa238f7-b340-4e88-b88f-29120c68a52a" containerName="payload-processing" containerID="cri-o://26960ac4fe0b5c4ec3892d8c6d9394201dd18284054ea9cd2fde581a04923fa8" gracePeriod=25
Oct 30 21:55:01.572389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.572354    6695 generic.go:356] "Generic (PLEG): container finished" podID="3aa238f7-b340-4e88-b88f-29120c68a52a" containerID="26960ac4fe0b5c4ec3892d8c6d9394201dd18284054ea9cd2fde581a04923fa8" exitCode=0
Oct 30 21:55:01.572560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.572428    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs" event={"ID":"3aa238f7-b340-4e88-b88f-29120c68a52a","Type":"ContainerDied","Data":"26960ac4fe0b5c4ec3892d8c6d9394201dd18284054ea9cd2fde581a04923fa8"}
Oct 30 21:55:01.753823 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.753795    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:55:01.769674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.769637    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r\") pod \"3aa238f7-b340-4e88-b88f-29120c68a52a\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") "
Oct 30 21:55:01.769832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.769786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials\") pod \"3aa238f7-b340-4e88-b88f-29120c68a52a\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") "
Oct 30 21:55:01.769832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.769815    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume\") pod \"3aa238f7-b340-4e88-b88f-29120c68a52a\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") "
Oct 30 21:55:01.769983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.769965    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume\") pod \"3aa238f7-b340-4e88-b88f-29120c68a52a\" (UID: \"3aa238f7-b340-4e88-b88f-29120c68a52a\") "
Oct 30 21:55:01.770327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.770245    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume" pod "3aa238f7-b340-4e88-b88f-29120c68a52a" (UID: "3aa238f7-b340-4e88-b88f-29120c68a52a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:01.770446 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.770323    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume" pod "3aa238f7-b340-4e88-b88f-29120c68a52a" (UID: "3aa238f7-b340-4e88-b88f-29120c68a52a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:01.770713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.770687    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:01.770713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.770715    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3aa238f7-b340-4e88-b88f-29120c68a52a-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:01.772107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.772082    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r" pod "3aa238f7-b340-4e88-b88f-29120c68a52a" (UID: "3aa238f7-b340-4e88-b88f-29120c68a52a"). InnerVolumeSpecName "kube-api-access-k2p2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:01.772186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.772139    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials" pod "3aa238f7-b340-4e88-b88f-29120c68a52a" (UID: "3aa238f7-b340-4e88-b88f-29120c68a52a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:01.871677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.871640    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2p2r\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-kube-api-access-k2p2r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:01.871677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:01.871675    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3aa238f7-b340-4e88-b88f-29120c68a52a-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:02.582855 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:02.582821    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"
Oct 30 21:55:02.583395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:02.582859    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs" event={"ID":"3aa238f7-b340-4e88-b88f-29120c68a52a","Type":"ContainerDied","Data":"a3a47ae998da3d5c3c29f9f2b1e3b3f5b72390fa545529b9c9ebdf72f9f70a2f"}
Oct 30 21:55:02.583395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:02.582931    6695 scope.go:122] "RemoveContainer" containerID="26960ac4fe0b5c4ec3892d8c6d9394201dd18284054ea9cd2fde581a04923fa8"
Oct 30 21:55:02.635031 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:02.634992    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"]
Oct 30 21:55:02.646592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:02.646556    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5g6fhs"]
Oct 30 21:55:03.253787 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:03.253748    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa238f7-b340-4e88-b88f-29120c68a52a" path="/var/lib/kubelet/pods/3aa238f7-b340-4e88-b88f-29120c68a52a/volumes"
Oct 30 21:55:03.951715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:03.951679    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"]
Oct 30 21:55:04.932754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:04.932718    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"]
Oct 30 21:55:04.943761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:04.943718    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"]
Oct 30 21:55:04.947965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:04.947940    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:55:08.952868 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:08.952813    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g" podUID="3765b785-1b74-4e11-b756-b023f8754c99" containerName="payload-processing" containerID="cri-o://73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e" gracePeriod=25
Oct 30 21:55:09.233580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.233553    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:55:09.293837 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.293805    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume\") pod \"3765b785-1b74-4e11-b756-b023f8754c99\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") "
Oct 30 21:55:09.294062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.294005    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh\") pod \"3765b785-1b74-4e11-b756-b023f8754c99\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") "
Oct 30 21:55:09.294062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.294032    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials\") pod \"3765b785-1b74-4e11-b756-b023f8754c99\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") "
Oct 30 21:55:09.294201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.294111    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume\") pod \"3765b785-1b74-4e11-b756-b023f8754c99\" (UID: \"3765b785-1b74-4e11-b756-b023f8754c99\") "
Oct 30 21:55:09.294337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.294302    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume" pod "3765b785-1b74-4e11-b756-b023f8754c99" (UID: "3765b785-1b74-4e11-b756-b023f8754c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:09.294801 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.294776    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:09.294920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.294881    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume" pod "3765b785-1b74-4e11-b756-b023f8754c99" (UID: "3765b785-1b74-4e11-b756-b023f8754c99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:09.296973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.296937    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials" pod "3765b785-1b74-4e11-b756-b023f8754c99" (UID: "3765b785-1b74-4e11-b756-b023f8754c99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:09.297097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.297071    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh" pod "3765b785-1b74-4e11-b756-b023f8754c99" (UID: "3765b785-1b74-4e11-b756-b023f8754c99"). InnerVolumeSpecName "kube-api-access-8vwlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:09.396522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.396483    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwlh\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-kube-api-access-8vwlh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:09.396522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.396517    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3765b785-1b74-4e11-b756-b023f8754c99-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:09.396522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.396527    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3765b785-1b74-4e11-b756-b023f8754c99-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:09.645095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.645061    6695 generic.go:356] "Generic (PLEG): container finished" podID="3765b785-1b74-4e11-b756-b023f8754c99" containerID="73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e" exitCode=0
Oct 30 21:55:09.645297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.645106    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"
Oct 30 21:55:09.645297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.645127    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g" event={"ID":"3765b785-1b74-4e11-b756-b023f8754c99","Type":"ContainerDied","Data":"73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e"}
Oct 30 21:55:09.645297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.645187    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g" event={"ID":"3765b785-1b74-4e11-b756-b023f8754c99","Type":"ContainerDied","Data":"8665d9e234bf915e926a71a437d5d3be9daefe46826371b31e3f9ba56cc3cfe6"}
Oct 30 21:55:09.645297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.645209    6695 scope.go:122] "RemoveContainer" containerID="73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e"
Oct 30 21:55:09.657030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.657005    6695 scope.go:122] "RemoveContainer" containerID="73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e"
Oct 30 21:55:09.657354 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:09.657332    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e\": container with ID starting with 73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e not found: ID does not exist" containerID="73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e"
Oct 30 21:55:09.657416 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.657363    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e"} err="failed to get container status \"73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e\": rpc error: code = NotFound desc = could not find container \"73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e\": container with ID starting with 73d19c1e19ce3edb451b530d6535e4e9432b57600b54dc9b9e3262120d2ea02e not found: ID does not exist"
Oct 30 21:55:09.695512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.695474    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"]
Oct 30 21:55:09.706979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.706945    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-75d9k2k4g"]
Oct 30 21:55:09.933768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.933666    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c" podUID="efd707cf-0987-49ae-a504-7dcd1226c050" containerName="payload-processing" containerID="cri-o://c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688" gracePeriod=25
Oct 30 21:55:09.944545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:09.944498    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx" podUID="8dd981eb-4e34-4364-9dab-b6e578a51ef1" containerName="payload-processing" containerID="cri-o://f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06" gracePeriod=25
Oct 30 21:55:10.236080 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.236054    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:55:10.287017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.286988    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:55:10.317723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.317691    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume\") pod \"efd707cf-0987-49ae-a504-7dcd1226c050\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") "
Oct 30 21:55:10.317931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.317803    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume\") pod \"efd707cf-0987-49ae-a504-7dcd1226c050\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") "
Oct 30 21:55:10.317931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.317843    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials\") pod \"efd707cf-0987-49ae-a504-7dcd1226c050\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") "
Oct 30 21:55:10.318118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.318099    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf\") pod \"efd707cf-0987-49ae-a504-7dcd1226c050\" (UID: \"efd707cf-0987-49ae-a504-7dcd1226c050\") "
Oct 30 21:55:10.318186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.318136    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume" pod "efd707cf-0987-49ae-a504-7dcd1226c050" (UID: "efd707cf-0987-49ae-a504-7dcd1226c050"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:10.318243 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.318207    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume" pod "efd707cf-0987-49ae-a504-7dcd1226c050" (UID: "efd707cf-0987-49ae-a504-7dcd1226c050"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:10.318858 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.318832    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.319279 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.319259    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd707cf-0987-49ae-a504-7dcd1226c050-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.320511 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.320487    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials" pod "efd707cf-0987-49ae-a504-7dcd1226c050" (UID: "efd707cf-0987-49ae-a504-7dcd1226c050"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:10.321262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.321247    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf" pod "efd707cf-0987-49ae-a504-7dcd1226c050" (UID: "efd707cf-0987-49ae-a504-7dcd1226c050"). InnerVolumeSpecName "kube-api-access-p5stf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:10.420683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.420647    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume\") pod \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") "
Oct 30 21:55:10.420877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.420713    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs\") pod \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") "
Oct 30 21:55:10.420877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.420776    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume\") pod \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") "
Oct 30 21:55:10.420877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.420824    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials\") pod \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\" (UID: \"8dd981eb-4e34-4364-9dab-b6e578a51ef1\") "
Oct 30 21:55:10.421171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.421140    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume" pod "8dd981eb-4e34-4364-9dab-b6e578a51ef1" (UID: "8dd981eb-4e34-4364-9dab-b6e578a51ef1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:10.421287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.421241    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume" pod "8dd981eb-4e34-4364-9dab-b6e578a51ef1" (UID: "8dd981eb-4e34-4364-9dab-b6e578a51ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:10.421636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.421586    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.421694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.421634    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5stf\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-kube-api-access-p5stf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.421694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.421650    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd981eb-4e34-4364-9dab-b6e578a51ef1-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.421694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.421663    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/efd707cf-0987-49ae-a504-7dcd1226c050-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.423027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.423001    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs" pod "8dd981eb-4e34-4364-9dab-b6e578a51ef1" (UID: "8dd981eb-4e34-4364-9dab-b6e578a51ef1"). InnerVolumeSpecName "kube-api-access-ws2xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:10.423121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.423021    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials" pod "8dd981eb-4e34-4364-9dab-b6e578a51ef1" (UID: "8dd981eb-4e34-4364-9dab-b6e578a51ef1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:10.523878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.523842    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.523878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.523875    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2xs\" (UniqueName: \"kubernetes.io/projected/8dd981eb-4e34-4364-9dab-b6e578a51ef1-kube-api-access-ws2xs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:10.655137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.655097    6695 generic.go:356] "Generic (PLEG): container finished" podID="8dd981eb-4e34-4364-9dab-b6e578a51ef1" containerID="f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06" exitCode=0
Oct 30 21:55:10.655362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.655148    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"
Oct 30 21:55:10.655362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.655189    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx" event={"ID":"8dd981eb-4e34-4364-9dab-b6e578a51ef1","Type":"ContainerDied","Data":"f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06"}
Oct 30 21:55:10.655362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.655281    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx" event={"ID":"8dd981eb-4e34-4364-9dab-b6e578a51ef1","Type":"ContainerDied","Data":"eed576bc2fd51a5f671b6e5e51b84ead5265e95a037b05d6f437ea26ad0d682f"}
Oct 30 21:55:10.655362 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.655313    6695 scope.go:122] "RemoveContainer" containerID="f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06"
Oct 30 21:55:10.656940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.656913    6695 generic.go:356] "Generic (PLEG): container finished" podID="efd707cf-0987-49ae-a504-7dcd1226c050" containerID="c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688" exitCode=0
Oct 30 21:55:10.657069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.656976    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"
Oct 30 21:55:10.657069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.656982    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c" event={"ID":"efd707cf-0987-49ae-a504-7dcd1226c050","Type":"ContainerDied","Data":"c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688"}
Oct 30 21:55:10.657069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.657021    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c" event={"ID":"efd707cf-0987-49ae-a504-7dcd1226c050","Type":"ContainerDied","Data":"32cc8ba5e4e8298ae4f507d5ba3aa7553fb3af2881845e7a5375ef8e201b2ce2"}
Oct 30 21:55:10.668931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.668911    6695 scope.go:122] "RemoveContainer" containerID="f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06"
Oct 30 21:55:10.669248 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:10.669227    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06\": container with ID starting with f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06 not found: ID does not exist" containerID="f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06"
Oct 30 21:55:10.669315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.669256    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06"} err="failed to get container status \"f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06\": rpc error: code = NotFound desc = could not find container \"f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06\": container with ID starting with f81ced85d71da55dc701ded9832ef319002dfcb85b28d44dd87cfff63d14ed06 not found: ID does not exist"
Oct 30 21:55:10.669315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.669273    6695 scope.go:122] "RemoveContainer" containerID="c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688"
Oct 30 21:55:10.680158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.680140    6695 scope.go:122] "RemoveContainer" containerID="c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688"
Oct 30 21:55:10.680472 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:10.680447    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688\": container with ID starting with c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688 not found: ID does not exist" containerID="c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688"
Oct 30 21:55:10.680581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.680477    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688"} err="failed to get container status \"c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688\": rpc error: code = NotFound desc = could not find container \"c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688\": container with ID starting with c8800a10e72036c80308e24fec3ed6a47c89db7ad4b9a5b71a3b584193394688 not found: ID does not exist"
Oct 30 21:55:10.711633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.711576    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"]
Oct 30 21:55:10.723177 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.723145    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5kb4nx"]
Oct 30 21:55:10.757550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.757518    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"]
Oct 30 21:55:10.762178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:10.762149    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6d67v4t5c"]
Oct 30 21:55:11.215461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:11.215429    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3765b785-1b74-4e11-b756-b023f8754c99" path="/var/lib/kubelet/pods/3765b785-1b74-4e11-b756-b023f8754c99/volumes"
Oct 30 21:55:11.215957 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:11.215941    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd981eb-4e34-4364-9dab-b6e578a51ef1" path="/var/lib/kubelet/pods/8dd981eb-4e34-4364-9dab-b6e578a51ef1/volumes"
Oct 30 21:55:11.216350 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:11.216337    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd707cf-0987-49ae-a504-7dcd1226c050" path="/var/lib/kubelet/pods/efd707cf-0987-49ae-a504-7dcd1226c050/volumes"
Oct 30 21:55:20.094712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.094678    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:55:20.108234 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.108206    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 21:55:20.113915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.113883    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"]
Oct 30 21:55:20.114721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114703    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7c1763a0-fc71-4e23-889b-0649db2465ea" containerName="revoke-keys"
Oct 30 21:55:20.114721 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114720    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1763a0-fc71-4e23-889b-0649db2465ea" containerName="revoke-keys"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114731    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3765b785-1b74-4e11-b756-b023f8754c99" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114736    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3765b785-1b74-4e11-b756-b023f8754c99" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114747    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8dd981eb-4e34-4364-9dab-b6e578a51ef1" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114753    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd981eb-4e34-4364-9dab-b6e578a51ef1" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114769    6695 cpu_manager.go:414] "containerMap: removing container" podUID="efd707cf-0987-49ae-a504-7dcd1226c050" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114774    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd707cf-0987-49ae-a504-7dcd1226c050" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114791    6695 cpu_manager.go:414] "containerMap: removing container" podUID="15548e1b-fc98-4ba4-8a15-d2cc85159602" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114796    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15548e1b-fc98-4ba4-8a15-d2cc85159602" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114813    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114818    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114841    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bfa21fc2-dcc1-4a5f-a628-e630065b872f" containerName="revoke-keys"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114845    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa21fc2-dcc1-4a5f-a628-e630065b872f" containerName="revoke-keys"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114854    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3aa238f7-b340-4e88-b88f-29120c68a52a" containerName="payload-processing"
Oct 30 21:55:20.114892 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114859    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa238f7-b340-4e88-b88f-29120c68a52a" containerName="payload-processing"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114986    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa21fc2-dcc1-4a5f-a628-e630065b872f" containerName="revoke-keys"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.114997    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa238f7-b340-4e88-b88f-29120c68a52a" containerName="payload-processing"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.115015    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1763a0-fc71-4e23-889b-0649db2465ea" containerName="revoke-keys"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.115027    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3765b785-1b74-4e11-b756-b023f8754c99" containerName="payload-processing"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.115037    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd981eb-4e34-4364-9dab-b6e578a51ef1" containerName="payload-processing"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.115044    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd707cf-0987-49ae-a504-7dcd1226c050" containerName="payload-processing"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.115052    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="15548e1b-fc98-4ba4-8a15-d2cc85159602" containerName="payload-processing"
Oct 30 21:55:20.115388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.115059    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0577fc-2bfe-4c97-a08e-5f7b12ce32f8" containerName="payload-processing"
Oct 30 21:55:20.119017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.118996    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.147030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.147001    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8d814a2cf4e\""
Oct 30 21:55:20.147030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.147000    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:55:20.147030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.147007    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:55:20.147030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.147042    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:55:20.147030 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.147013    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rxxkz\""
Oct 30 21:55:20.147380 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.147048    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:55:20.161810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.161782    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"]
Oct 30 21:55:20.225882 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.225849    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"]
Oct 30 21:55:20.231115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.231092    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.255852 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.255823    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5d3601-openshift-default-dockercfg-4s794\""
Oct 30 21:55:20.293704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.293657    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.294186 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.294144    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r496p\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.320665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.317965    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.320665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.318074    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.419108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419010    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419108 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419094    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"]
Oct 30 21:55:20.419348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419129    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419241    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r496p\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.419348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419327    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419366    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419397    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419525 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419458    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419521    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.419727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419585    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.419727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419660    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbggj\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.419918 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419893    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.419989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.419971    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.420057 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.420037    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.421745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.420906    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.421745 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.420926    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.450153 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.450122    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.461977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.461942    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r496p\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p\") pod \"payload-processing-external-model-686b96ccfc-wk285\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") " pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:20.520908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.520870    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521002    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521051    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521072    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521114    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521226    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbggj\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521308    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521335    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521336    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521416    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521425    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521742    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.521807 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.521784    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.522255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.522227    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.523719 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.523694    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.523970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.523952    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.542312 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.542282    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.542520 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.542498    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbggj\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj\") pod \"e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") " pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.547255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.547227    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:20.734033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.733942    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:21.000993 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:20.960246    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"]
Oct 30 21:55:21.034478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.034018    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:55:21.034478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.034086    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:55:21.034478 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.034114    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:55:21.256220 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.256193    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"]
Oct 30 21:55:21.258257 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:55:21.258226    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6586ff36_960d_4541_a273_96223d5446a5.slice/crio-5a334e0bdd514f096464d4e44e3700fdcc80d3d682f634db4c9cb86c87546652 WatchSource:0}: Error finding container 5a334e0bdd514f096464d4e44e3700fdcc80d3d682f634db4c9cb86c87546652: Status 404 returned error can't find the container with id 5a334e0bdd514f096464d4e44e3700fdcc80d3d682f634db4c9cb86c87546652
Oct 30 21:55:21.772980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.772802    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285" event={"ID":"6586ff36-960d-4541-a273-96223d5446a5","Type":"ContainerStarted","Data":"fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076"}
Oct 30 21:55:21.772980 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.772927    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285" event={"ID":"6586ff36-960d-4541-a273-96223d5446a5","Type":"ContainerStarted","Data":"5a334e0bdd514f096464d4e44e3700fdcc80d3d682f634db4c9cb86c87546652"}
Oct 30 21:55:21.773896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.773000    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:21.775863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.775826    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" event={"ID":"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe","Type":"ContainerStarted","Data":"8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5"}
Oct 30 21:55:21.775978 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.775889    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" event={"ID":"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe","Type":"ContainerStarted","Data":"3d256396b1bdec4c517fe388dee57cb0f521b318df114ac575d075a096960710"}
Oct 30 21:55:21.820584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.820515    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285" podStartSLOduration=1.820496172 podStartE2EDuration="1.820496172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:21.817650063 +0000 UTC m=+2704.740108362" watchObservedRunningTime="2026-10-30 21:55:21.820496172 +0000 UTC m=+2704.742954470"
Oct 30 21:55:21.865664 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:21.865578    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podStartSLOduration=1.865564555 podStartE2EDuration="1.865564555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:21.865229402 +0000 UTC m=+2704.787687700" watchObservedRunningTime="2026-10-30 21:55:21.865564555 +0000 UTC m=+2704.788022853"
Oct 30 21:55:22.345064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.345015    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"]
Oct 30 21:55:22.350238 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.350212    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.370616 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.370569    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3d5d13\""
Oct 30 21:55:22.370778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.370631    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"routing-overlay\""
Oct 30 21:55:22.370778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.370753    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"openshift-service-ca.crt\""
Oct 30 21:55:22.371833 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.371812    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"]
Oct 30 21:55:22.373257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.373243    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"kube-root-ca.crt\""
Oct 30 21:55:22.374061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.374049    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"agc-cred-e899723f90d2\""
Oct 30 21:55:22.378258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.378242    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"payload-processing-external-model-e2e--4472330f-dockercfg-ld5jc\""
Oct 30 21:55:22.437554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.437116    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.437554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.437176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.437554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.437252    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmh4\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.437554 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.437342    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.548425 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.548384    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.548973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.548943    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.549134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.549118    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.549308 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.549289    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmh4\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.560626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.550740    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.585804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.562793    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.586066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.563450    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:22.586223 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.574688    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:55:22.586390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.586359    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused"
Oct 30 21:55:22.586803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.578321    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmh4\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.631708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.631245    6695 scope.go:122] "RemoveContainer" containerID="9fb96343730b5af210914cb6956b09b3065c282cd309ed80117d7bdd226ad6fc"
Oct 30 21:55:22.652349 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.633838    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:22.681182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:22.678089    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:23.041744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.041701    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"]
Oct 30 21:55:23.048470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.048441    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.058755 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.058721    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.058919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.058837    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.058919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.058891    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.059007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.058977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gfw\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.075746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.075712    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"]
Oct 30 21:55:23.084198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.084150    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"agc-cred-8ce500f40c94\""
Oct 30 21:55:23.085445 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.084673    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"]
Oct 30 21:55:23.090175 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:55:23.090148    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526167ad_21d4_4908_8481_94c4ef66c8a6.slice/crio-2330638d3aed16e66a814d001d6f22d33a0fc04c2880a0ec5b936590017233d1 WatchSource:0}: Error finding container 2330638d3aed16e66a814d001d6f22d33a0fc04c2880a0ec5b936590017233d1: Status 404 returned error can't find the container with id 2330638d3aed16e66a814d001d6f22d33a0fc04c2880a0ec5b936590017233d1
Oct 30 21:55:23.160262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.160223    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.160419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.160345    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.160419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.160379    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.162774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.160448    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gfw\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.162774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.161693    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.162774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.162189    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.163908 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.163883    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.180414 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.180384    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gfw\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.367876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.367777    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:23.548402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.548360    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:55:23.548648 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.548417    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused"
Oct 30 21:55:23.735775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.731784    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"]
Oct 30 21:55:23.736078 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:55:23.736030    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3014f8c4_83ea_41c2_b04b_417f6b69a764.slice/crio-b397386b9faf0c543aba114ac24570a4e55a872c8d8283a818a2715c0b459835 WatchSource:0}: Error finding container b397386b9faf0c543aba114ac24570a4e55a872c8d8283a818a2715c0b459835: Status 404 returned error can't find the container with id b397386b9faf0c543aba114ac24570a4e55a872c8d8283a818a2715c0b459835
Oct 30 21:55:23.804593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.804547    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr" event={"ID":"526167ad-21d4-4908-8481-94c4ef66c8a6","Type":"ContainerStarted","Data":"fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418"}
Oct 30 21:55:23.804768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.804641    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:23.804768 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.804669    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr" event={"ID":"526167ad-21d4-4908-8481-94c4ef66c8a6","Type":"ContainerStarted","Data":"2330638d3aed16e66a814d001d6f22d33a0fc04c2880a0ec5b936590017233d1"}
Oct 30 21:55:23.806162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.806133    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w" event={"ID":"3014f8c4-83ea-41c2-b04b-417f6b69a764","Type":"ContainerStarted","Data":"b397386b9faf0c543aba114ac24570a4e55a872c8d8283a818a2715c0b459835"}
Oct 30 21:55:23.865064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:23.865009    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr" podStartSLOduration=1.864996157 podStartE2EDuration="1.864996157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:23.860130504 +0000 UTC m=+2706.782588812" watchObservedRunningTime="2026-10-30 21:55:23.864996157 +0000 UTC m=+2706.787454461"
Oct 30 21:55:24.549007 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:24.548974    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:55:24.549399 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:24.549026    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused"
Oct 30 21:55:24.841750 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:24.841660    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w" event={"ID":"3014f8c4-83ea-41c2-b04b-417f6b69a764","Type":"ContainerStarted","Data":"aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb"}
Oct 30 21:55:24.841915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:24.841758    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:24.880700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:24.880644    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w" podStartSLOduration=2.880630238 podStartE2EDuration="2.880630238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:24.87892119 +0000 UTC m=+2707.801379489" watchObservedRunningTime="2026-10-30 21:55:24.880630238 +0000 UTC m=+2707.803088537"
Oct 30 21:55:25.547773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:25.547736    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:55:25.547976 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:25.547799    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused"
Oct 30 21:55:26.548620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:26.548576    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:55:26.549158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:26.548661    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused"
Oct 30 21:55:27.549178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:27.549131    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body=
Oct 30 21:55:27.550274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:27.549886    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused"
Oct 30 21:55:28.650494 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:28.650460    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:28.651385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:28.651358    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:28.652292 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:28.652267    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 21:55:31.075185 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:31.075144    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:55:33.810532 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:33.810498    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:55:34.845221 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:34.845188    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:36.862654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:36.862621    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:55:37.004681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:37.004640    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"]
Oct 30 21:55:42.005040 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.004993    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr" podUID="526167ad-21d4-4908-8481-94c4ef66c8a6" containerName="payload-processing" containerID="cri-o://fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418" gracePeriod=25
Oct 30 21:55:42.284182 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.284154    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:42.366793 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.366753    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4\") pod \"526167ad-21d4-4908-8481-94c4ef66c8a6\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") "
Oct 30 21:55:42.367009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.366925    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials\") pod \"526167ad-21d4-4908-8481-94c4ef66c8a6\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") "
Oct 30 21:55:42.367009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.366983    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume\") pod \"526167ad-21d4-4908-8481-94c4ef66c8a6\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") "
Oct 30 21:55:42.367146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.367085    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume\") pod \"526167ad-21d4-4908-8481-94c4ef66c8a6\" (UID: \"526167ad-21d4-4908-8481-94c4ef66c8a6\") "
Oct 30 21:55:42.367904 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.367646    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume" pod "526167ad-21d4-4908-8481-94c4ef66c8a6" (UID: "526167ad-21d4-4908-8481-94c4ef66c8a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:42.368144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.368122    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:42.368757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.368102    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume" pod "526167ad-21d4-4908-8481-94c4ef66c8a6" (UID: "526167ad-21d4-4908-8481-94c4ef66c8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:42.369659 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.369593    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4" pod "526167ad-21d4-4908-8481-94c4ef66c8a6" (UID: "526167ad-21d4-4908-8481-94c4ef66c8a6"). InnerVolumeSpecName "kube-api-access-8vmh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:42.370949 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.370898    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials" pod "526167ad-21d4-4908-8481-94c4ef66c8a6" (UID: "526167ad-21d4-4908-8481-94c4ef66c8a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:42.469917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.469881    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmh4\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-kube-api-access-8vmh4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:42.469917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.469911    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/526167ad-21d4-4908-8481-94c4ef66c8a6-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:42.469917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:42.469923    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/526167ad-21d4-4908-8481-94c4ef66c8a6-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:43.107061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.107019    6695 generic.go:356] "Generic (PLEG): container finished" podID="526167ad-21d4-4908-8481-94c4ef66c8a6" containerID="fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418" exitCode=0
Oct 30 21:55:43.107582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.107085    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"
Oct 30 21:55:43.107582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.107106    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr" event={"ID":"526167ad-21d4-4908-8481-94c4ef66c8a6","Type":"ContainerDied","Data":"fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418"}
Oct 30 21:55:43.107582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.107172    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr" event={"ID":"526167ad-21d4-4908-8481-94c4ef66c8a6","Type":"ContainerDied","Data":"2330638d3aed16e66a814d001d6f22d33a0fc04c2880a0ec5b936590017233d1"}
Oct 30 21:55:43.107582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.107205    6695 scope.go:122] "RemoveContainer" containerID="fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418"
Oct 30 21:55:43.121829 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.121803    6695 scope.go:122] "RemoveContainer" containerID="fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418"
Oct 30 21:55:43.122169 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:43.122143    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418\": container with ID starting with fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418 not found: ID does not exist" containerID="fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418"
Oct 30 21:55:43.122305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.122174    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418"} err="failed to get container status \"fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418\": rpc error: code = NotFound desc = could not find container \"fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418\": container with ID starting with fa84c066c160d0d01ad1d61f86701ae52aa8e511eb26683e20b5a7cdb7658418 not found: ID does not exist"
Oct 30 21:55:43.164392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.164358    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"]
Oct 30 21:55:43.175519 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.175488    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99ht5nr"]
Oct 30 21:55:43.216977 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.216945    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="526167ad-21d4-4908-8481-94c4ef66c8a6" path="/var/lib/kubelet/pods/526167ad-21d4-4908-8481-94c4ef66c8a6/volumes"
Oct 30 21:55:43.369475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.369391    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"]
Oct 30 21:55:43.370477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.370448    6695 cpu_manager.go:414] "containerMap: removing container" podUID="526167ad-21d4-4908-8481-94c4ef66c8a6" containerName="payload-processing"
Oct 30 21:55:43.370477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.370473    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="526167ad-21d4-4908-8481-94c4ef66c8a6" containerName="payload-processing"
Oct 30 21:55:43.370705 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.370657    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="526167ad-21d4-4908-8481-94c4ef66c8a6" containerName="payload-processing"
Oct 30 21:55:43.375687 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.375665    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.390042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.389987    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:43.390042 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.390028    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:55:43.390285 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.390081    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:55:43.390487 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.390469    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:55:43.393551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.393525    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"]
Oct 30 21:55:43.482444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.482402    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4xt\" (UniqueName: \"kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.482709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.482686    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.482786 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.482734    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.584545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.584507    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.584761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.584563    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.584761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.584697    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4xt\" (UniqueName: \"kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.585405 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.585373    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.587219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.587197    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.606675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.606642    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4xt\" (UniqueName: \"kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt\") pod \"maas-api-revoke-sub-ae23572b-bq2m7\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:43.690399 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:43.690316    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:44.027832 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:44.027787    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"]
Oct 30 21:55:44.032914 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:55:44.032886    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae41f69_7450_4b7c_b65a_fe872e9bd4ac.slice/crio-21cc66634a9b7ab0378dd7127545c3ca47b3d2c17ed8d6ed3960aea2a79ecbc3 WatchSource:0}: Error finding container 21cc66634a9b7ab0378dd7127545c3ca47b3d2c17ed8d6ed3960aea2a79ecbc3: Status 404 returned error can't find the container with id 21cc66634a9b7ab0378dd7127545c3ca47b3d2c17ed8d6ed3960aea2a79ecbc3
Oct 30 21:55:44.119759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:44.119722    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7" event={"ID":"bae41f69-7450-4b7c-b65a-fe872e9bd4ac","Type":"ContainerStarted","Data":"21cc66634a9b7ab0378dd7127545c3ca47b3d2c17ed8d6ed3960aea2a79ecbc3"}
Oct 30 21:55:45.130491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:45.130451    6695 generic.go:356] "Generic (PLEG): container finished" podID="bae41f69-7450-4b7c-b65a-fe872e9bd4ac" containerID="1ea665ea2bdc0ad135a02f927f12122b5507eb87a326bdbe5d6b60fccf120690" exitCode=0
Oct 30 21:55:45.130948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:45.130593    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7" event={"ID":"bae41f69-7450-4b7c-b65a-fe872e9bd4ac","Type":"ContainerDied","Data":"1ea665ea2bdc0ad135a02f927f12122b5507eb87a326bdbe5d6b60fccf120690"}
Oct 30 21:55:45.793946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:45.793913    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"]
Oct 30 21:55:46.679490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.679388    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:46.773999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.773952    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt\" (UniqueName: \"kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt\") pod \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") "
Oct 30 21:55:46.774226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.774054    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca\") pod \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") "
Oct 30 21:55:46.774226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.774149    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca\") pod \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\" (UID: \"bae41f69-7450-4b7c-b65a-fe872e9bd4ac\") "
Oct 30 21:55:46.775322 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.775287    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca" pod "bae41f69-7450-4b7c-b65a-fe872e9bd4ac" (UID: "bae41f69-7450-4b7c-b65a-fe872e9bd4ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.778740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.778327    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt" pod "bae41f69-7450-4b7c-b65a-fe872e9bd4ac" (UID: "bae41f69-7450-4b7c-b65a-fe872e9bd4ac"). InnerVolumeSpecName "kube-api-access-ml4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:46.778740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.779175    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca" pod "bae41f69-7450-4b7c-b65a-fe872e9bd4ac" (UID: "bae41f69-7450-4b7c-b65a-fe872e9bd4ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:46.876878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.876778    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4xt\" (UniqueName: \"kubernetes.io/projected/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-kube-api-access-ml4xt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:46.876878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.876824    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:46.876878 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:46.876840    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae41f69-7450-4b7c-b65a-fe872e9bd4ac-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:47.170683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:47.167488    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"
Oct 30 21:55:47.170683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:47.167562    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7" event={"ID":"bae41f69-7450-4b7c-b65a-fe872e9bd4ac","Type":"ContainerDied","Data":"21cc66634a9b7ab0378dd7127545c3ca47b3d2c17ed8d6ed3960aea2a79ecbc3"}
Oct 30 21:55:47.170683 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:47.167636    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21cc66634a9b7ab0378dd7127545c3ca47b3d2c17ed8d6ed3960aea2a79ecbc3"
Oct 30 21:55:50.794226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:50.794144    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8" podUID="ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" containerName="payload-processing" containerID="cri-o://c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26" gracePeriod=25
Oct 30 21:55:51.096764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.096733    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:55:51.207827 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.207794    6695 generic.go:356] "Generic (PLEG): container finished" podID="ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" containerID="c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26" exitCode=0
Oct 30 21:55:51.208043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.207907    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"
Oct 30 21:55:51.220166 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.220133    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8" event={"ID":"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26","Type":"ContainerDied","Data":"c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26"}
Oct 30 21:55:51.220328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.220188    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8" event={"ID":"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26","Type":"ContainerDied","Data":"a9fe454772f13fcce8be001b60c19780fc831e35faf10ea90904460783060bad"}
Oct 30 21:55:51.220328 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.220215    6695 scope.go:122] "RemoveContainer" containerID="c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26"
Oct 30 21:55:51.231256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.231234    6695 scope.go:122] "RemoveContainer" containerID="c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26"
Oct 30 21:55:51.231591 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:55:51.231570    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26\": container with ID starting with c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26 not found: ID does not exist" containerID="c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26"
Oct 30 21:55:51.231674 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.231615    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26"} err="failed to get container status \"c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26\": rpc error: code = NotFound desc = could not find container \"c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26\": container with ID starting with c9f377b014dfe5949753cfd7e30c2c9f75c6108ac2026e38a31b692618124f26 not found: ID does not exist"
Oct 30 21:55:51.255104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.255072    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials\") pod \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") "
Oct 30 21:55:51.255256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.255129    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h\") pod \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") "
Oct 30 21:55:51.255256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.255198    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume\") pod \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") "
Oct 30 21:55:51.255256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.255248    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume\") pod \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\" (UID: \"ed07da6b-5f23-4e2c-88a2-0ab5bb633d26\") "
Oct 30 21:55:51.255712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.255578    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume" pod "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" (UID: "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:51.255813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.255728    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume" pod "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" (UID: "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:51.256273 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.256250    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:51.256321 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.256283    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:51.257518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.257497    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h" pod "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" (UID: "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26"). InnerVolumeSpecName "kube-api-access-r8l9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:51.257592 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.257530    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials" pod "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" (UID: "ed07da6b-5f23-4e2c-88a2-0ab5bb633d26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:51.357417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.357323    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:51.357417 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.357359    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8l9h\" (UniqueName: \"kubernetes.io/projected/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26-kube-api-access-r8l9h\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:55:51.560111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.560074    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"]
Oct 30 21:55:51.572044 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:51.572014    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d852jrq8"]
Oct 30 21:55:53.216489 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:53.216454    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" path="/var/lib/kubelet/pods/ed07da6b-5f23-4e2c-88a2-0ab5bb633d26/volumes"
Oct 30 21:55:54.688970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:54.688935    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"]
Oct 30 21:55:58.645159 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.645124    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"]
Oct 30 21:55:58.645996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.645970    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bae41f69-7450-4b7c-b65a-fe872e9bd4ac" containerName="revoke-keys"
Oct 30 21:55:58.645996 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.645990    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae41f69-7450-4b7c-b65a-fe872e9bd4ac" containerName="revoke-keys"
Oct 30 21:55:58.646171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.646015    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" containerName="payload-processing"
Oct 30 21:55:58.646171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.646022    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" containerName="payload-processing"
Oct 30 21:55:58.646171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.646151    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae41f69-7450-4b7c-b65a-fe872e9bd4ac" containerName="revoke-keys"
Oct 30 21:55:58.646171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.646164    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed07da6b-5f23-4e2c-88a2-0ab5bb633d26" containerName="payload-processing"
Oct 30 21:55:58.651370 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.651347    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.670174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.670145    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"]
Oct 30 21:55:58.676797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.676761    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"agc-cred-2f214c160f30\""
Oct 30 21:55:58.676797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.676762    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"payload-processing-external-model-e2e--d53c5c12-dockercfg-h66cr\""
Oct 30 21:55:58.714539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.714502    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.714810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.714773    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl52r\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.714895 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.714880    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.714937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.714915    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.817624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.816583    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl52r\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.817624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.816743    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.817624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.816794    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.817624 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.816875    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.820351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.820321    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.821036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.821012    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.834258 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.834221    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.854104 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.849379    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl52r\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:58.966803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:58.966717    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:55:59.307518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:59.307486    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"]
Oct 30 21:55:59.310911 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:55:59.310879    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfbb711_ffe2_4ffe_a6c5_7ff9b9f15c42.slice/crio-790a23e27abfa507dcdadd5f9b484d22674651d3a0b5d9dee0f97a1bfbb79b6a WatchSource:0}: Error finding container 790a23e27abfa507dcdadd5f9b484d22674651d3a0b5d9dee0f97a1bfbb79b6a: Status 404 returned error can't find the container with id 790a23e27abfa507dcdadd5f9b484d22674651d3a0b5d9dee0f97a1bfbb79b6a
Oct 30 21:55:59.364544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:59.364506    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt" event={"ID":"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42","Type":"ContainerStarted","Data":"790a23e27abfa507dcdadd5f9b484d22674651d3a0b5d9dee0f97a1bfbb79b6a"}
Oct 30 21:55:59.689779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:59.689669    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5" podUID="3d600247-d43d-48d4-904b-f79f6cd02872" containerName="payload-processing" containerID="cri-o://bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f" gracePeriod=25
Oct 30 21:55:59.987618 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:55:59.987572    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:56:00.047502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.047459    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume\") pod \"3d600247-d43d-48d4-904b-f79f6cd02872\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") "
Oct 30 21:56:00.048375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.048345    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume" pod "3d600247-d43d-48d4-904b-f79f6cd02872" (UID: "3d600247-d43d-48d4-904b-f79f6cd02872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:00.149119 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.149083    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials\") pod \"3d600247-d43d-48d4-904b-f79f6cd02872\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") "
Oct 30 21:56:00.149320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.149139    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume\") pod \"3d600247-d43d-48d4-904b-f79f6cd02872\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") "
Oct 30 21:56:00.149320 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.149171    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv\") pod \"3d600247-d43d-48d4-904b-f79f6cd02872\" (UID: \"3d600247-d43d-48d4-904b-f79f6cd02872\") "
Oct 30 21:56:00.149535 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.149508    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume" pod "3d600247-d43d-48d4-904b-f79f6cd02872" (UID: "3d600247-d43d-48d4-904b-f79f6cd02872"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:00.149789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.149769    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:00.149789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.149787    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d600247-d43d-48d4-904b-f79f6cd02872-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:00.151505 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.151480    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials" pod "3d600247-d43d-48d4-904b-f79f6cd02872" (UID: "3d600247-d43d-48d4-904b-f79f6cd02872"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:00.151638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.151508    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv" pod "3d600247-d43d-48d4-904b-f79f6cd02872" (UID: "3d600247-d43d-48d4-904b-f79f6cd02872"). InnerVolumeSpecName "kube-api-access-5nvqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:00.250839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.250745    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:00.250839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.250778    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nvqv\" (UniqueName: \"kubernetes.io/projected/3d600247-d43d-48d4-904b-f79f6cd02872-kube-api-access-5nvqv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:00.374433 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.374399    6695 generic.go:356] "Generic (PLEG): container finished" podID="3d600247-d43d-48d4-904b-f79f6cd02872" containerID="bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f" exitCode=0
Oct 30 21:56:00.374690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.374457    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"
Oct 30 21:56:00.374690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.374480    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5" event={"ID":"3d600247-d43d-48d4-904b-f79f6cd02872","Type":"ContainerDied","Data":"bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f"}
Oct 30 21:56:00.374690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.374537    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5" event={"ID":"3d600247-d43d-48d4-904b-f79f6cd02872","Type":"ContainerDied","Data":"dffce6108fa3d030b54fed935b949a9ccfca1b67c82dfac96d23b0db78f5c6c5"}
Oct 30 21:56:00.374690 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.374560    6695 scope.go:122] "RemoveContainer" containerID="bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f"
Oct 30 21:56:00.376538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.376510    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt" event={"ID":"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42","Type":"ContainerStarted","Data":"8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c"}
Oct 30 21:56:00.376699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.376569    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:56:00.386825 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.386806    6695 scope.go:122] "RemoveContainer" containerID="bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f"
Oct 30 21:56:00.387060 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:00.387043    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f\": container with ID starting with bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f not found: ID does not exist" containerID="bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f"
Oct 30 21:56:00.387115 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.387068    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f"} err="failed to get container status \"bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f\": rpc error: code = NotFound desc = could not find container \"bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f\": container with ID starting with bb6ecaa5d6277bfbf09418519ad4343f5cfc73698ceff2ae1e9dc2718151fa4f not found: ID does not exist"
Oct 30 21:56:00.428480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.428444    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"]
Oct 30 21:56:00.432812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.432782    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-84fbhrqp5"]
Oct 30 21:56:00.468544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:00.468479    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt" podStartSLOduration=2.468462067 podStartE2EDuration="2.468462067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:00.466132906 +0000 UTC m=+2743.388591203" watchObservedRunningTime="2026-10-30 21:56:00.468462067 +0000 UTC m=+2743.390920367"
Oct 30 21:56:01.076253 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:01.076216    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:56:01.248633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:01.217750    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d600247-d43d-48d4-904b-f79f6cd02872" path="/var/lib/kubelet/pods/3d600247-d43d-48d4-904b-f79f6cd02872/volumes"
Oct 30 21:56:11.390181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:11.390150    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:56:15.916944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:15.916904    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"]
Oct 30 21:56:15.918116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:15.918088    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3d600247-d43d-48d4-904b-f79f6cd02872" containerName="payload-processing"
Oct 30 21:56:15.918116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:15.918113    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d600247-d43d-48d4-904b-f79f6cd02872" containerName="payload-processing"
Oct 30 21:56:15.918367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:15.918352    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d600247-d43d-48d4-904b-f79f6cd02872" containerName="payload-processing"
Oct 30 21:56:15.924329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:15.924303    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:15.954217 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:15.954184    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"]
Oct 30 21:56:16.095764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.095724    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.095764 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.095766    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.095870    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.095917    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.095996    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.096116    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.096144    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.096191    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drng\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.096365 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.096341    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.197658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197549    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.197658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197624    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.197874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197680    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drng\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.197874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197751    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.197874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197851    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197883    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197942    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.197977    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198046 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.198020    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.198049    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198247 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.198201    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.198349    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.198520    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.198702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.198554    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.200367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.200346    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.200501 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.200482    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.218253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.218222    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.218407 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.218222    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drng\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng\") pod \"maas-default-gateway-openshift-default-77f556d647-rc9jm\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.243305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.243272    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:16.569698 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.569656    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"]
Oct 30 21:56:16.570967 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:16.570935    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e053cb_05bf_4a0c_8749_0871878e6a35.slice/crio-3e597d004893017d9e23c625e85968beeca48494c7786f4195c9bb8373331db6 WatchSource:0}: Error finding container 3e597d004893017d9e23c625e85968beeca48494c7786f4195c9bb8373331db6: Status 404 returned error can't find the container with id 3e597d004893017d9e23c625e85968beeca48494c7786f4195c9bb8373331db6
Oct 30 21:56:16.574886 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.574849    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:16.574965 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.574923    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:16.575009 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:16.574964    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:17.537326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:17.537272    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" event={"ID":"b4e053cb-05bf-4a0c-8749-0871878e6a35","Type":"ContainerStarted","Data":"fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a"}
Oct 30 21:56:17.538036 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:17.537992    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" event={"ID":"b4e053cb-05bf-4a0c-8749-0871878e6a35","Type":"ContainerStarted","Data":"3e597d004893017d9e23c625e85968beeca48494c7786f4195c9bb8373331db6"}
Oct 30 21:56:17.637508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:17.634821    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podStartSLOduration=2.6347967260000003 podStartE2EDuration="2.634796726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:17.596664693 +0000 UTC m=+2760.519122987" watchObservedRunningTime="2026-10-30 21:56:17.634796726 +0000 UTC m=+2760.557255025"
Oct 30 21:56:18.244432 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:18.244391    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:18.246123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:18.246091    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:18.246264 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:18.246156    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:19.244074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:19.244040    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:19.244584 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:19.244112    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:20.244464 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:20.244419    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:20.244923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:20.244477    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:21.244355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:21.244312    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:21.244757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:21.244721    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:22.244171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:22.244125    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:22.244402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:22.244206    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:23.244442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:23.244342    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:23.244442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:23.244420    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:24.243702 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:24.243665    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused" start-of-body=
Oct 30 21:56:24.243919 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:24.243725    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": dial tcp 10.134.1.115:15021: connect: connection refused"
Oct 30 21:56:25.384270 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.384235    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"]
Oct 30 21:56:25.389122 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.389096    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.395495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.395467    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.395615 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.395506    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.395803 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.395781    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.408059 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.408039    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:25.408162 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.408147    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:25.419742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.419709    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:25.420761 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.420729    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:56:25.435893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.435860    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"]
Oct 30 21:56:25.496726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.496690    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.496916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.496791    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.496916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.496815    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.497670 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.497636    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.499920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.499898    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.517542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.517513    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr\") pod \"maas-api-revoke-sub-65a67d34-2l8rs\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.708261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.708168    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:25.812712 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.811174    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:25.812966 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.812825    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:25.819481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:25.814545    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 21:56:26.037590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:26.037562    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"]
Oct 30 21:56:26.041178 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:26.041141    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2332cd_30fd_407f_9646_41786134fc7c.slice/crio-b1396bd6b0549c10353c1220c36e5018d622e7830fc35dc5dfa4cf5e68dd2984 WatchSource:0}: Error finding container b1396bd6b0549c10353c1220c36e5018d622e7830fc35dc5dfa4cf5e68dd2984: Status 404 returned error can't find the container with id b1396bd6b0549c10353c1220c36e5018d622e7830fc35dc5dfa4cf5e68dd2984
Oct 30 21:56:26.629810 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:26.629773    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs" event={"ID":"1a2332cd-30fd-407f-9646-41786134fc7c","Type":"ContainerStarted","Data":"dbce35fc3892e5eab973b0bdd144b8ab73ac6a42ebed57c0ccc27f2f97123b91"}
Oct 30 21:56:26.630225 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:26.629832    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs" event={"ID":"1a2332cd-30fd-407f-9646-41786134fc7c","Type":"ContainerStarted","Data":"b1396bd6b0549c10353c1220c36e5018d622e7830fc35dc5dfa4cf5e68dd2984"}
Oct 30 21:56:26.685686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:26.685516    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs" podStartSLOduration=1.6854971490000001 podStartE2EDuration="1.685497149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:26.683286342 +0000 UTC m=+2769.605744639" watchObservedRunningTime="2026-10-30 21:56:26.685497149 +0000 UTC m=+2769.607955448"
Oct 30 21:56:27.639396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:27.639359    6695 generic.go:356] "Generic (PLEG): container finished" podID="1a2332cd-30fd-407f-9646-41786134fc7c" containerID="dbce35fc3892e5eab973b0bdd144b8ab73ac6a42ebed57c0ccc27f2f97123b91" exitCode=0
Oct 30 21:56:27.639851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:27.639444    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs" event={"ID":"1a2332cd-30fd-407f-9646-41786134fc7c","Type":"ContainerDied","Data":"dbce35fc3892e5eab973b0bdd144b8ab73ac6a42ebed57c0ccc27f2f97123b91"}
Oct 30 21:56:28.848325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.848293    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:28.860025 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.860001    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca\") pod \"1a2332cd-30fd-407f-9646-41786134fc7c\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") "
Oct 30 21:56:28.860233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.860219    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca\") pod \"1a2332cd-30fd-407f-9646-41786134fc7c\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") "
Oct 30 21:56:28.860278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.860266    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr\") pod \"1a2332cd-30fd-407f-9646-41786134fc7c\" (UID: \"1a2332cd-30fd-407f-9646-41786134fc7c\") "
Oct 30 21:56:28.860354 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.860333    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca" pod "1a2332cd-30fd-407f-9646-41786134fc7c" (UID: "1a2332cd-30fd-407f-9646-41786134fc7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:28.860641 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.860588    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:28.862329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.862307    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr" pod "1a2332cd-30fd-407f-9646-41786134fc7c" (UID: "1a2332cd-30fd-407f-9646-41786134fc7c"). InnerVolumeSpecName "kube-api-access-mt4tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:28.862406 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.862307    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca" pod "1a2332cd-30fd-407f-9646-41786134fc7c" (UID: "1a2332cd-30fd-407f-9646-41786134fc7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:28.961662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.961564    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2332cd-30fd-407f-9646-41786134fc7c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:28.961662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:28.961596    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt4tr\" (UniqueName: \"kubernetes.io/projected/1a2332cd-30fd-407f-9646-41786134fc7c-kube-api-access-mt4tr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:29.659851 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:29.659819    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"
Oct 30 21:56:29.660043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:29.659847    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs" event={"ID":"1a2332cd-30fd-407f-9646-41786134fc7c","Type":"ContainerDied","Data":"b1396bd6b0549c10353c1220c36e5018d622e7830fc35dc5dfa4cf5e68dd2984"}
Oct 30 21:56:29.660043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:29.659901    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1396bd6b0549c10353c1220c36e5018d622e7830fc35dc5dfa4cf5e68dd2984"
Oct 30 21:56:30.978116 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:30.978072    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"]
Oct 30 21:56:30.979154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:30.979104    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1a2332cd-30fd-407f-9646-41786134fc7c" containerName="revoke-keys"
Oct 30 21:56:30.979154 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:30.979127    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2332cd-30fd-407f-9646-41786134fc7c" containerName="revoke-keys"
Oct 30 21:56:30.980401 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:30.979334    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2332cd-30fd-407f-9646-41786134fc7c" containerName="revoke-keys"
Oct 30 21:56:30.998890 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:30.998859    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.019206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.019169    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"]
Oct 30 21:56:31.082874 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:31.082833    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:56:31.096181 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096147    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096194    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096393 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096299    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096507    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096581 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096562    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096706    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096784 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096771    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096839 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096804    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.096917 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.096894    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.198774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.198732    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.198958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.198783    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.198958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.198816    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.198958 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.198928    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.198975    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199026    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199061    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199095    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199130    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199189    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199376    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199408    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199501    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.199614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.199581    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.201240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.201214    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.201528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.201510    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.218429 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.218398    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.218579 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.218561    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx\") pod \"maas-default-gateway-openshift-default-77f556d647-fj2hm\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.322813 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.322769    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:31.672652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.672618    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"]
Oct 30 21:56:31.674285 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:31.674255    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e13172_cb33_402f_b875_5a1991357f13.slice/crio-e215d677105c8e25a1dcc5235e3f179f55e19e8215d93b283e1626f326df70d3 WatchSource:0}: Error finding container e215d677105c8e25a1dcc5235e3f179f55e19e8215d93b283e1626f326df70d3: Status 404 returned error can't find the container with id e215d677105c8e25a1dcc5235e3f179f55e19e8215d93b283e1626f326df70d3
Oct 30 21:56:31.677379 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.677346    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:31.677466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.677417    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:31.677466 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:31.677448    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:32.688339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:32.688300    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" event={"ID":"78e13172-cb33-402f-b875-5a1991357f13","Type":"ContainerStarted","Data":"db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e"}
Oct 30 21:56:32.688911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:32.688377    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" event={"ID":"78e13172-cb33-402f-b875-5a1991357f13","Type":"ContainerStarted","Data":"e215d677105c8e25a1dcc5235e3f179f55e19e8215d93b283e1626f326df70d3"}
Oct 30 21:56:32.764923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:32.764846    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podStartSLOduration=2.764820699 podStartE2EDuration="2.764820699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:32.749239616 +0000 UTC m=+2775.671697914" watchObservedRunningTime="2026-10-30 21:56:32.764820699 +0000 UTC m=+2775.687278998"
Oct 30 21:56:33.324023 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:33.323987    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:33.341806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:33.341766    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:33.342476 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:33.342255    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:34.324100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:34.324061    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:34.324752 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:34.324716    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:35.331626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:35.324024    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:35.331626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:35.324120    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:36.333626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:36.332728    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:36.333626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:36.332791    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:37.113628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.111364    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"]
Oct 30 21:56:37.280172 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.280137    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.291734 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.291703    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"]
Oct 30 21:56:37.309265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309231    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"openshift-service-ca.crt\""
Oct 30 21:56:37.309645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309466    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"kube-root-ca.crt\""
Oct 30 21:56:37.309645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309499    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"payload-processing-external-model-e2e--c4b58ec6-dockercfg-hxwmh\""
Oct 30 21:56:37.309645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309498    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"agc-cred-68bf0c456c5f\""
Oct 30 21:56:37.309645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309577    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"e2e-cfl-3e7cf6-provider-cred\""
Oct 30 21:56:37.309645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309622    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"routing-overlay\""
Oct 30 21:56:37.309930 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.309708    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-805edc\""
Oct 30 21:56:37.337630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.334597    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:37.337630 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.334685    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:37.338211 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.337656    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"]
Oct 30 21:56:37.344499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.344470    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.349431 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.349409    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"]
Oct 30 21:56:37.399278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.399162    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:37.399278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.399166    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:37.399278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.399209    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:37.399522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.399305    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:56:37.464915 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.464874    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.465106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.465044    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.465106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.465090    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.465300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.465276    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.465491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.465468    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9l4\" (UniqueName: \"kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.465575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.465561    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brlj\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.465778 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.465757    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.567513 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567476    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.567757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567533    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9l4\" (UniqueName: \"kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.567757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567593    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brlj\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.567757 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567728    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.567974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567778    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.567974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567864    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.567974 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.567912    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.568538 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.568509    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.568686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.568596    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.568796 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.568768    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.570458 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.570428    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.570620 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.570578    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.586704 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.586676    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brlj\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.586846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.586803    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9l4\" (UniqueName: \"kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4\") pod \"maas-api-revoke-sub-a679259f-gxqjb\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.600863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.600823    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:37.665970 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.665654    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:37.968772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:37.968740    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"]
Oct 30 21:56:37.970558 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:37.970529    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e33497_2b0f_49e2_a8ef_98e2d6a2c898.slice/crio-24937bed293ca6361001245f80aa895108d668f93a0e20b04639243b2a8406c6 WatchSource:0}: Error finding container 24937bed293ca6361001245f80aa895108d668f93a0e20b04639243b2a8406c6: Status 404 returned error can't find the container with id 24937bed293ca6361001245f80aa895108d668f93a0e20b04639243b2a8406c6
Oct 30 21:56:38.040657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.040614    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"]
Oct 30 21:56:38.055717 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:38.055687    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7012197_6304_44e7_881b_ba0a661a5aa0.slice/crio-6ea2d9452714718da6895b2a9ed7ba90a8a79cdb3349e5e95f8e79ad2745f8d9 WatchSource:0}: Error finding container 6ea2d9452714718da6895b2a9ed7ba90a8a79cdb3349e5e95f8e79ad2745f8d9: Status 404 returned error can't find the container with id 6ea2d9452714718da6895b2a9ed7ba90a8a79cdb3349e5e95f8e79ad2745f8d9
Oct 30 21:56:38.323744 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.323712    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:38.323940 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.323761    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:38.785012 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.784972    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" event={"ID":"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898","Type":"ContainerStarted","Data":"77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a"}
Oct 30 21:56:38.785441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.785051    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:38.785441 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.785076    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" event={"ID":"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898","Type":"ContainerStarted","Data":"24937bed293ca6361001245f80aa895108d668f93a0e20b04639243b2a8406c6"}
Oct 30 21:56:38.787123 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.787094    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb" event={"ID":"f7012197-6304-44e7-881b-ba0a661a5aa0","Type":"ContainerStarted","Data":"b97038c9901ead34e195994fbc70ec70bd501ee5ba6e3439d125e12f9133eef4"}
Oct 30 21:56:38.787257 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.787144    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb" event={"ID":"f7012197-6304-44e7-881b-ba0a661a5aa0","Type":"ContainerStarted","Data":"6ea2d9452714718da6895b2a9ed7ba90a8a79cdb3349e5e95f8e79ad2745f8d9"}
Oct 30 21:56:38.827363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.827310    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" podStartSLOduration=1.827294054 podStartE2EDuration="1.827294054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:38.822456787 +0000 UTC m=+2781.744915088" watchObservedRunningTime="2026-10-30 21:56:38.827294054 +0000 UTC m=+2781.749752349"
Oct 30 21:56:38.864461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:38.864387    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb" podStartSLOduration=1.8643645599999998 podStartE2EDuration="1.86436456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:38.857802803 +0000 UTC m=+2781.780261102" watchObservedRunningTime="2026-10-30 21:56:38.86436456 +0000 UTC m=+2781.786822858"
Oct 30 21:56:39.323804 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:39.323758    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body=
Oct 30 21:56:39.324000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:39.323827    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused"
Oct 30 21:56:39.798136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:39.798100    6695 generic.go:356] "Generic (PLEG): container finished" podID="f7012197-6304-44e7-881b-ba0a661a5aa0" containerID="b97038c9901ead34e195994fbc70ec70bd501ee5ba6e3439d125e12f9133eef4" exitCode=0
Oct 30 21:56:39.798677 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:39.798187    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb" event={"ID":"f7012197-6304-44e7-881b-ba0a661a5aa0","Type":"ContainerDied","Data":"b97038c9901ead34e195994fbc70ec70bd501ee5ba6e3439d125e12f9133eef4"}
Oct 30 21:56:41.092275 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.092219    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:41.093790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.093761    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:41.094287 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.094265    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:41.095043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.095019    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 21:56:41.242681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.242585    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca\") pod \"f7012197-6304-44e7-881b-ba0a661a5aa0\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") "
Oct 30 21:56:41.242681 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.242657    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4\" (UniqueName: \"kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4\") pod \"f7012197-6304-44e7-881b-ba0a661a5aa0\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") "
Oct 30 21:56:41.242939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.242719    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca\") pod \"f7012197-6304-44e7-881b-ba0a661a5aa0\" (UID: \"f7012197-6304-44e7-881b-ba0a661a5aa0\") "
Oct 30 21:56:41.243146 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.243118    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca" pod "f7012197-6304-44e7-881b-ba0a661a5aa0" (UID: "f7012197-6304-44e7-881b-ba0a661a5aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:41.243785 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.243756    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:41.244899 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.244871    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca" pod "f7012197-6304-44e7-881b-ba0a661a5aa0" (UID: "f7012197-6304-44e7-881b-ba0a661a5aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:41.245148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.245119    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4" pod "f7012197-6304-44e7-881b-ba0a661a5aa0" (UID: "f7012197-6304-44e7-881b-ba0a661a5aa0"). InnerVolumeSpecName "kube-api-access-nr9l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:41.345158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.345123    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7012197-6304-44e7-881b-ba0a661a5aa0-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:41.345158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.345160    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9l4\" (UniqueName: \"kubernetes.io/projected/f7012197-6304-44e7-881b-ba0a661a5aa0-kube-api-access-nr9l4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:41.857640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.857596    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"
Oct 30 21:56:41.857640 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.857635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb" event={"ID":"f7012197-6304-44e7-881b-ba0a661a5aa0","Type":"ContainerDied","Data":"6ea2d9452714718da6895b2a9ed7ba90a8a79cdb3349e5e95f8e79ad2745f8d9"}
Oct 30 21:56:41.857900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:41.857683    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea2d9452714718da6895b2a9ed7ba90a8a79cdb3349e5e95f8e79ad2745f8d9"
Oct 30 21:56:49.801782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.801750    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:49.888804 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:49.888705    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-68bf0c456c5f: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:49.888804 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:49.888745    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:49.889105 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:49.888829    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials podName:a0e33497-2b0f-49e2-a8ef-98e2d6a2c898 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:50.388808217 +0000 UTC m=+2793.311266512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898") : secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:49.906838 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.906801    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"]
Oct 30 21:56:49.946971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.946917    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"]
Oct 30 21:56:49.948254 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.948233    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f7012197-6304-44e7-881b-ba0a661a5aa0" containerName="revoke-keys"
Oct 30 21:56:49.948383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.948266    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7012197-6304-44e7-881b-ba0a661a5aa0" containerName="revoke-keys"
Oct 30 21:56:49.948435 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.948425    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7012197-6304-44e7-881b-ba0a661a5aa0" containerName="revoke-keys"
Oct 30 21:56:49.952709 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.952686    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:49.976551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:49.976515    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"]
Oct 30 21:56:50.094680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.093619    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.094680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.093805    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv2bk\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.094680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.093949    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.094680 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.094041    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.161440 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.161386    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"]
Oct 30 21:56:50.184638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.184002    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.189167 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.189134    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"]
Oct 30 21:56:50.197939 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.197906    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv2bk\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.209144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.199326    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:50.209144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.199338    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:56:50.209144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.199372    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:50.209144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.199377    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:50.209480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.209229    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.209480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.209338    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.209480 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.209460    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.227824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.211063    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.227824 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.211899    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.229746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.229530    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv2bk\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.230537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.230496    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.269652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.269617    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.314237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.314125    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.314237 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.314217    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.314566 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.314549    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg492\" (UniqueName: \"kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.424766 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.424721    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.424988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.424866    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.425095 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.425049    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg492\" (UniqueName: \"kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.426390 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.426359    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.426517 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:50.426491    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-68bf0c456c5f: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:50.426590 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:50.426519    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:50.426682 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:50.426612    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials podName:a0e33497-2b0f-49e2-a8ef-98e2d6a2c898 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:51.426579804 +0000 UTC m=+2794.349038100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898") : secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:50.433409 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.433377    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.453068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.452950    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg492\" (UniqueName: \"kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492\") pod \"maas-api-revoke-sub-0632f1f3-x7mrg\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.510461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.510420    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:50.601987 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:50.601954    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3414edb7_7dae_493b_b3d2_884dc5ae2329.slice/crio-d21d4b7e95c89c13d527d060e86fd8dbc3746ef5575edbbc381461db7cc60580 WatchSource:0}: Error finding container d21d4b7e95c89c13d527d060e86fd8dbc3746ef5575edbbc381461db7cc60580: Status 404 returned error can't find the container with id d21d4b7e95c89c13d527d060e86fd8dbc3746ef5575edbbc381461db7cc60580
Oct 30 21:56:50.617672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.617641    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"]
Oct 30 21:56:50.830053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.830024    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"]
Oct 30 21:56:50.832833 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:56:50.832805    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99eb095b_0823_4fd4_8c18_8cc1680ffc71.slice/crio-0fe1c68cfd01a0412d5e322ebe87fcc69d52bd5cb216dbe6e8aae98bca5229f0 WatchSource:0}: Error finding container 0fe1c68cfd01a0412d5e322ebe87fcc69d52bd5cb216dbe6e8aae98bca5229f0: Status 404 returned error can't find the container with id 0fe1c68cfd01a0412d5e322ebe87fcc69d52bd5cb216dbe6e8aae98bca5229f0
Oct 30 21:56:50.937518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.937470    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg" event={"ID":"99eb095b-0823-4fd4-8c18-8cc1680ffc71","Type":"ContainerStarted","Data":"0fe1c68cfd01a0412d5e322ebe87fcc69d52bd5cb216dbe6e8aae98bca5229f0"}
Oct 30 21:56:50.939324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.939296    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv" event={"ID":"3414edb7-7dae-493b-b3d2-884dc5ae2329","Type":"ContainerStarted","Data":"489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6"}
Oct 30 21:56:50.939481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.939347    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv" event={"ID":"3414edb7-7dae-493b-b3d2-884dc5ae2329","Type":"ContainerStarted","Data":"d21d4b7e95c89c13d527d060e86fd8dbc3746ef5575edbbc381461db7cc60580"}
Oct 30 21:56:50.939481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.939449    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:56:50.979198 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:50.979140    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv" podStartSLOduration=1.9791252049999999 podStartE2EDuration="1.979125205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:50.978518717 +0000 UTC m=+2793.900977016" watchObservedRunningTime="2026-10-30 21:56:50.979125205 +0000 UTC m=+2793.901583503"
Oct 30 21:56:51.441662 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:51.441623    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-68bf0c456c5f: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:51.441662 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:51.441672    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:51.441905 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:51.441723    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials podName:a0e33497-2b0f-49e2-a8ef-98e2d6a2c898 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:53.441709997 +0000 UTC m=+2796.364168276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898") : secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:51.950378 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:51.950336    6695 generic.go:356] "Generic (PLEG): container finished" podID="99eb095b-0823-4fd4-8c18-8cc1680ffc71" containerID="a311063393769708c7fdf88ddbc392a678612692081e5f9090e60a77771ec8d6" exitCode=0
Oct 30 21:56:51.950922 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:51.950423    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg" event={"ID":"99eb095b-0823-4fd4-8c18-8cc1680ffc71","Type":"ContainerDied","Data":"a311063393769708c7fdf88ddbc392a678612692081e5f9090e60a77771ec8d6"}
Oct 30 21:56:53.156428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.156396    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:53.277190 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.277154    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492\" (UniqueName: \"kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492\") pod \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") "
Oct 30 21:56:53.277387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.277217    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca\") pod \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") "
Oct 30 21:56:53.277387 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.277351    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca\") pod \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\" (UID: \"99eb095b-0823-4fd4-8c18-8cc1680ffc71\") "
Oct 30 21:56:53.277765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.277737    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca" pod "99eb095b-0823-4fd4-8c18-8cc1680ffc71" (UID: "99eb095b-0823-4fd4-8c18-8cc1680ffc71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:53.278068 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.278047    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:53.279375 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.279347    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca" pod "99eb095b-0823-4fd4-8c18-8cc1680ffc71" (UID: "99eb095b-0823-4fd4-8c18-8cc1680ffc71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:53.279455 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.279386    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492" pod "99eb095b-0823-4fd4-8c18-8cc1680ffc71" (UID: "99eb095b-0823-4fd4-8c18-8cc1680ffc71"). InnerVolumeSpecName "kube-api-access-sg492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:53.379216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.379179    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg492\" (UniqueName: \"kubernetes.io/projected/99eb095b-0823-4fd4-8c18-8cc1680ffc71-kube-api-access-sg492\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:53.379216 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.379215    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99eb095b-0823-4fd4-8c18-8cc1680ffc71-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:53.480704 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:53.480667    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-68bf0c456c5f: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:53.480704 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:53.480702    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z: secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:53.480930 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:53.480758    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials podName:a0e33497-2b0f-49e2-a8ef-98e2d6a2c898 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:57.480740952 +0000 UTC m=+2800.403199230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898") : secret "agc-cred-68bf0c456c5f" not found
Oct 30 21:56:53.968463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.968430    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"
Oct 30 21:56:53.968730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.968465    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg" event={"ID":"99eb095b-0823-4fd4-8c18-8cc1680ffc71","Type":"ContainerDied","Data":"0fe1c68cfd01a0412d5e322ebe87fcc69d52bd5cb216dbe6e8aae98bca5229f0"}
Oct 30 21:56:53.968730 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:53.968513    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe1c68cfd01a0412d5e322ebe87fcc69d52bd5cb216dbe6e8aae98bca5229f0"
Oct 30 21:56:54.928746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:54.928703    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" podUID="a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" containerName="payload-processing" containerID="cri-o://77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a" gracePeriod=25
Oct 30 21:56:55.220256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.220225    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:55.310867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.310816    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj\") pod \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") "
Oct 30 21:56:55.311161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.310898    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume\") pod \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") "
Oct 30 21:56:55.311161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.310964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume\") pod \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") "
Oct 30 21:56:55.311404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.311377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume" pod "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:55.311678 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.311653    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials\") pod \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\" (UID: \"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898\") "
Oct 30 21:56:55.312368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.312341    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:55.312368 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.311656    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume" pod "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:55.313647 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.313589    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj" pod "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898"). InnerVolumeSpecName "kube-api-access-2brlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:55.314176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.314150    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials" pod "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" (UID: "a0e33497-2b0f-49e2-a8ef-98e2d6a2c898"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:55.413797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.413762    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:55.413797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.413792    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2brlj\" (UniqueName: \"kubernetes.io/projected/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-kube-api-access-2brlj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:55.413797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.413803    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:56:55.988239 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.988198    6695 generic.go:356] "Generic (PLEG): container finished" podID="a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" containerID="77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a" exitCode=0
Oct 30 21:56:55.988770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.988249    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"
Oct 30 21:56:55.988770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.988249    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" event={"ID":"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898","Type":"ContainerDied","Data":"77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a"}
Oct 30 21:56:55.988770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.988323    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z" event={"ID":"a0e33497-2b0f-49e2-a8ef-98e2d6a2c898","Type":"ContainerDied","Data":"24937bed293ca6361001245f80aa895108d668f93a0e20b04639243b2a8406c6"}
Oct 30 21:56:55.988770 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:55.988360    6695 scope.go:122] "RemoveContainer" containerID="77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a"
Oct 30 21:56:56.007300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:56.007278    6695 scope.go:122] "RemoveContainer" containerID="77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a"
Oct 30 21:56:56.007599 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:56:56.007575    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a\": container with ID starting with 77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a not found: ID does not exist" containerID="77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a"
Oct 30 21:56:56.007749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:56.007625    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a"} err="failed to get container status \"77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a\": rpc error: code = NotFound desc = could not find container \"77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a\": container with ID starting with 77465bb08445acf3f36bc13fb4e48e545c7970b0d81283e5d891e4be9de3c55a not found: ID does not exist"
Oct 30 21:56:56.064117 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:56.064078    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"]
Oct 30 21:56:56.076011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:56.075978    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-6d859v96z"]
Oct 30 21:56:57.216549 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:56:57.216514    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" path="/var/lib/kubelet/pods/a0e33497-2b0f-49e2-a8ef-98e2d6a2c898/volumes"
Oct 30 21:57:01.227399 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:01.227202    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:57:01.954165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:01.954132    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:57:02.591971 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.591931    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"]
Oct 30 21:57:02.593048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.593017    6695 cpu_manager.go:414] "containerMap: removing container" podUID="99eb095b-0823-4fd4-8c18-8cc1680ffc71" containerName="revoke-keys"
Oct 30 21:57:02.593048 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.593047    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99eb095b-0823-4fd4-8c18-8cc1680ffc71" containerName="revoke-keys"
Oct 30 21:57:02.593256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.593123    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" containerName="payload-processing"
Oct 30 21:57:02.593256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.593132    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" containerName="payload-processing"
Oct 30 21:57:02.593364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.593338    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="99eb095b-0823-4fd4-8c18-8cc1680ffc71" containerName="revoke-keys"
Oct 30 21:57:02.593418 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.593371    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e33497-2b0f-49e2-a8ef-98e2d6a2c898" containerName="payload-processing"
Oct 30 21:57:02.599079 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.599043    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.618508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.618472    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"]
Oct 30 21:57:02.628651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.624078    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"agc-cred-5467e7a748c7\""
Oct 30 21:57:02.661746 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.661705    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.661935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.661753    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.662037 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.662004    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbglp\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.662111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.662049    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.763447 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.763402    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbglp\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.763707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.763468    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.763707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.763658    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.763707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.763693    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.764336 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.764305    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.764459 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.764431    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.766083 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.766062    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.782063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.782025    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbglp\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:02.915472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:02.915377    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:03.303210 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.294773    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"]
Oct 30 21:57:03.308276 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:03.308237    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891fe98c_6da1_4bd9_9d00_768831ba982f.slice/crio-7d6374a1ab64adc0fb2448fd8fd940a5761a9fca60495a998117314a7cb8dd03 WatchSource:0}: Error finding container 7d6374a1ab64adc0fb2448fd8fd940a5761a9fca60495a998117314a7cb8dd03: Status 404 returned error can't find the container with id 7d6374a1ab64adc0fb2448fd8fd940a5761a9fca60495a998117314a7cb8dd03
Oct 30 21:57:03.598591 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.598504    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"]
Oct 30 21:57:03.603500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.603472    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.617946 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.617916    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:03.622759 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.622727    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"]
Oct 30 21:57:03.628137 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.628111    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:03.628286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.628149    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:57:03.628286 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.628180    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:03.695516 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.695479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.695728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.695673    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.695834 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.695809    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58dn\" (UniqueName: \"kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.797540 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.797497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.797727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.797651    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.797819 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.797737    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58dn\" (UniqueName: \"kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.798329 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.798302    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.800078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.800051    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.817069 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.817040    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58dn\" (UniqueName: \"kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn\") pod \"maas-api-revoke-sub-cb403179-s25r6\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:03.921315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:03.921224    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:04.140174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:04.140125    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" event={"ID":"891fe98c-6da1-4bd9-9d00-768831ba982f","Type":"ContainerStarted","Data":"62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210"}
Oct 30 21:57:04.140389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:04.140187    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" event={"ID":"891fe98c-6da1-4bd9-9d00-768831ba982f","Type":"ContainerStarted","Data":"7d6374a1ab64adc0fb2448fd8fd940a5761a9fca60495a998117314a7cb8dd03"}
Oct 30 21:57:04.140389 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:04.140217    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:04.191962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:04.191855    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" podStartSLOduration=2.191836517 podStartE2EDuration="2.191836517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:04.190961909 +0000 UTC m=+2807.113420212" watchObservedRunningTime="2026-10-30 21:57:04.191836517 +0000 UTC m=+2807.114294816"
Oct 30 21:57:04.204186 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:04.204143    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:04.204186 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:04.204183    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:04.204423 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:04.204269    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials podName:891fe98c-6da1-4bd9-9d00-768831ba982f nodeName:}" failed. No retries permitted until 2026-10-30 21:57:04.704246841 +0000 UTC m=+2807.626705139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:04.247226 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:04.247190    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"]
Oct 30 21:57:04.251737 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:04.251708    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1f3a02_4f77_4290_b3c3_f1f6e32e308c.slice/crio-6ecb77ceedb0da4539dac8b642cb4a5c4ce7e866409cec470e2ef904e0e9a4ad WatchSource:0}: Error finding container 6ecb77ceedb0da4539dac8b642cb4a5c4ce7e866409cec470e2ef904e0e9a4ad: Status 404 returned error can't find the container with id 6ecb77ceedb0da4539dac8b642cb4a5c4ce7e866409cec470e2ef904e0e9a4ad
Oct 30 21:57:04.712939 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:04.712906    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:04.712939 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:04.712939    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:04.713382 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:04.713004    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials podName:891fe98c-6da1-4bd9-9d00-768831ba982f nodeName:}" failed. No retries permitted until 2026-10-30 21:57:05.712989842 +0000 UTC m=+2808.635448119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:05.036427 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.036387    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"]
Oct 30 21:57:05.046817 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.046759    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85a9eb00-xsjgq"]
Oct 30 21:57:05.161781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.161729    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6" event={"ID":"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c","Type":"ContainerStarted","Data":"10fbea976dc0a88c456eaefd1194e51de3630a4962e214453af2f94dd0ad1901"}
Oct 30 21:57:05.161781 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.161826    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6" event={"ID":"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c","Type":"ContainerStarted","Data":"6ecb77ceedb0da4539dac8b642cb4a5c4ce7e866409cec470e2ef904e0e9a4ad"}
Oct 30 21:57:05.204337 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.204280    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6" podStartSLOduration=2.204263055 podStartE2EDuration="2.204263055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:05.198283192 +0000 UTC m=+2808.120741490" watchObservedRunningTime="2026-10-30 21:57:05.204263055 +0000 UTC m=+2808.126721370"
Oct 30 21:57:05.218814 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.218782    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1259d0e-ff8e-46ae-ab11-b986ff33d51e" path="/var/lib/kubelet/pods/d1259d0e-ff8e-46ae-ab11-b986ff33d51e/volumes"
Oct 30 21:57:05.623294 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.623256    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"]
Oct 30 21:57:05.627906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:05.627873    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"]
Oct 30 21:57:05.735558 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:05.735521    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:05.735558 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:05.735552    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:05.736048 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:05.735634    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials podName:891fe98c-6da1-4bd9-9d00-768831ba982f nodeName:}" failed. No retries permitted until 2026-10-30 21:57:07.735587814 +0000 UTC m=+2810.658046090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:06.173334 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:06.173297    6695 generic.go:356] "Generic (PLEG): container finished" podID="7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" containerID="10fbea976dc0a88c456eaefd1194e51de3630a4962e214453af2f94dd0ad1901" exitCode=0
Oct 30 21:57:06.173534 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:06.173378    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6" event={"ID":"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c","Type":"ContainerDied","Data":"10fbea976dc0a88c456eaefd1194e51de3630a4962e214453af2f94dd0ad1901"}
Oct 30 21:57:07.385373 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.385336    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:07.469053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.468958    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn\" (UniqueName: \"kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn\") pod \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") "
Oct 30 21:57:07.469053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.468999    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca\") pod \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") "
Oct 30 21:57:07.469053 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.469053    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca\") pod \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\" (UID: \"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c\") "
Oct 30 21:57:07.469483 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.469452    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca" pod "7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" (UID: "7c1f3a02-4f77-4290-b3c3-f1f6e32e308c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:07.470002 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.469976    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.472553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.472390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca" pod "7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" (UID: "7c1f3a02-4f77-4290-b3c3-f1f6e32e308c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:07.472553 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.472449    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn" pod "7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" (UID: "7c1f3a02-4f77-4290-b3c3-f1f6e32e308c"). InnerVolumeSpecName "kube-api-access-d58dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:07.572261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.572227    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58dn\" (UniqueName: \"kubernetes.io/projected/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-kube-api-access-d58dn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.572261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:07.572258    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.775193 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:07.775158    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-805edc/agc-cred-5467e7a748c7: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:07.775193 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:07.775189    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt: secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:07.775441 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:07.775245    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials podName:891fe98c-6da1-4bd9-9d00-768831ba982f nodeName:}" failed. No retries permitted until 2026-10-30 21:57:11.775229463 +0000 UTC m=+2814.697687756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f") : secret "agc-cred-5467e7a748c7" not found
Oct 30 21:57:08.191972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:08.191889    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6" event={"ID":"7c1f3a02-4f77-4290-b3c3-f1f6e32e308c","Type":"ContainerDied","Data":"6ecb77ceedb0da4539dac8b642cb4a5c4ce7e866409cec470e2ef904e0e9a4ad"}
Oct 30 21:57:08.191972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:08.191938    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ecb77ceedb0da4539dac8b642cb4a5c4ce7e866409cec470e2ef904e0e9a4ad"
Oct 30 21:57:08.191972 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:08.191901    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"
Oct 30 21:57:10.624314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.624254    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv" podUID="3414edb7-7dae-493b-b3d2-884dc5ae2329" containerName="payload-processing" containerID="cri-o://489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6" gracePeriod=25
Oct 30 21:57:10.923400 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.923366    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:57:10.931676 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.931647    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk\") pod \"3414edb7-7dae-493b-b3d2-884dc5ae2329\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") "
Oct 30 21:57:10.931836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.931693    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume\") pod \"3414edb7-7dae-493b-b3d2-884dc5ae2329\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") "
Oct 30 21:57:10.931836 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.931768    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume\") pod \"3414edb7-7dae-493b-b3d2-884dc5ae2329\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") "
Oct 30 21:57:10.931938 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.931846    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials\") pod \"3414edb7-7dae-493b-b3d2-884dc5ae2329\" (UID: \"3414edb7-7dae-493b-b3d2-884dc5ae2329\") "
Oct 30 21:57:10.932251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.932171    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume" pod "3414edb7-7dae-493b-b3d2-884dc5ae2329" (UID: "3414edb7-7dae-493b-b3d2-884dc5ae2329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:10.932251 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.932185    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume" pod "3414edb7-7dae-493b-b3d2-884dc5ae2329" (UID: "3414edb7-7dae-493b-b3d2-884dc5ae2329"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:10.932524 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.932502    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:10.932590 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.932531    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3414edb7-7dae-493b-b3d2-884dc5ae2329-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:10.933989 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.933961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk" pod "3414edb7-7dae-493b-b3d2-884dc5ae2329" (UID: "3414edb7-7dae-493b-b3d2-884dc5ae2329"). InnerVolumeSpecName "kube-api-access-xv2bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:10.934103 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:10.934020    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials" pod "3414edb7-7dae-493b-b3d2-884dc5ae2329" (UID: "3414edb7-7dae-493b-b3d2-884dc5ae2329"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:11.033944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.033901    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv2bk\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-kube-api-access-xv2bk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:11.033944 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.033943    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3414edb7-7dae-493b-b3d2-884dc5ae2329-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:11.173831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.173725    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" podUID="891fe98c-6da1-4bd9-9d00-768831ba982f" containerName="payload-processing" containerID="cri-o://62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210" gracePeriod=25
Oct 30 21:57:11.218969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.218930    6695 generic.go:356] "Generic (PLEG): container finished" podID="3414edb7-7dae-493b-b3d2-884dc5ae2329" containerID="489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6" exitCode=0
Oct 30 21:57:11.219148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.218973    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"
Oct 30 21:57:11.219148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.218979    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv" event={"ID":"3414edb7-7dae-493b-b3d2-884dc5ae2329","Type":"ContainerDied","Data":"489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6"}
Oct 30 21:57:11.219148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.219035    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv" event={"ID":"3414edb7-7dae-493b-b3d2-884dc5ae2329","Type":"ContainerDied","Data":"d21d4b7e95c89c13d527d060e86fd8dbc3746ef5575edbbc381461db7cc60580"}
Oct 30 21:57:11.219148 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.219065    6695 scope.go:122] "RemoveContainer" containerID="489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6"
Oct 30 21:57:11.277343 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.277306    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"]
Oct 30 21:57:11.282064 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.282037    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-9cd5sm4nv"]
Oct 30 21:57:11.287536 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.287508    6695 scope.go:122] "RemoveContainer" containerID="489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6"
Oct 30 21:57:11.288044 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:11.288018    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6\": container with ID starting with 489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6 not found: ID does not exist" containerID="489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6"
Oct 30 21:57:11.288160 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.288049    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6"} err="failed to get container status \"489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6\": rpc error: code = NotFound desc = could not find container \"489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6\": container with ID starting with 489deb0873b03f8212a3e7498cf008ae8ad9393672d2bd6e0f9e11555701f1d6 not found: ID does not exist"
Oct 30 21:57:11.511582 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.511556    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:11.539863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.539827    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials\") pod \"891fe98c-6da1-4bd9-9d00-768831ba982f\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") "
Oct 30 21:57:11.540043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.539965    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp\") pod \"891fe98c-6da1-4bd9-9d00-768831ba982f\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") "
Oct 30 21:57:11.540128 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.540111    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume\") pod \"891fe98c-6da1-4bd9-9d00-768831ba982f\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") "
Oct 30 21:57:11.540193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.540176    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume\") pod \"891fe98c-6da1-4bd9-9d00-768831ba982f\" (UID: \"891fe98c-6da1-4bd9-9d00-768831ba982f\") "
Oct 30 21:57:11.540526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.540499    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume" pod "891fe98c-6da1-4bd9-9d00-768831ba982f" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:11.540526 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.540515    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume" pod "891fe98c-6da1-4bd9-9d00-768831ba982f" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:11.540846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.540820    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:11.540907 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.540850    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/891fe98c-6da1-4bd9-9d00-768831ba982f-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:11.542548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.542530    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials" pod "891fe98c-6da1-4bd9-9d00-768831ba982f" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:11.542636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.542556    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp" pod "891fe98c-6da1-4bd9-9d00-768831ba982f" (UID: "891fe98c-6da1-4bd9-9d00-768831ba982f"). InnerVolumeSpecName "kube-api-access-kbglp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:11.642249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.642213    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbglp\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-kube-api-access-kbglp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:11.642249 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:11.642247    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/891fe98c-6da1-4bd9-9d00-768831ba982f-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:12.232076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.232036    6695 generic.go:356] "Generic (PLEG): container finished" podID="891fe98c-6da1-4bd9-9d00-768831ba982f" containerID="62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210" exitCode=0
Oct 30 21:57:12.232326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.232090    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"
Oct 30 21:57:12.232326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.232104    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" event={"ID":"891fe98c-6da1-4bd9-9d00-768831ba982f","Type":"ContainerDied","Data":"62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210"}
Oct 30 21:57:12.232326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.232186    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt" event={"ID":"891fe98c-6da1-4bd9-9d00-768831ba982f","Type":"ContainerDied","Data":"7d6374a1ab64adc0fb2448fd8fd940a5761a9fca60495a998117314a7cb8dd03"}
Oct 30 21:57:12.232326 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.232217    6695 scope.go:122] "RemoveContainer" containerID="62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210"
Oct 30 21:57:12.246898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.246870    6695 scope.go:122] "RemoveContainer" containerID="62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210"
Oct 30 21:57:12.247217 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:12.247189    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210\": container with ID starting with 62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210 not found: ID does not exist" containerID="62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210"
Oct 30 21:57:12.247302 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.247229    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210"} err="failed to get container status \"62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210\": rpc error: code = NotFound desc = could not find container \"62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210\": container with ID starting with 62d387336d016a9fbcb8b8a057c6c8e4c5709e62d7fb50c22621632138dd5210 not found: ID does not exist"
Oct 30 21:57:12.286777 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.286744    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"]
Oct 30 21:57:12.297733 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:12.297697    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-774bcspbt"]
Oct 30 21:57:13.216751 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:13.216719    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3414edb7-7dae-493b-b3d2-884dc5ae2329" path="/var/lib/kubelet/pods/3414edb7-7dae-493b-b3d2-884dc5ae2329/volumes"
Oct 30 21:57:13.217233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:13.217218    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891fe98c-6da1-4bd9-9d00-768831ba982f" path="/var/lib/kubelet/pods/891fe98c-6da1-4bd9-9d00-768831ba982f/volumes"
Oct 30 21:57:14.285410 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.285375    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"]
Oct 30 21:57:14.286438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286412    6695 cpu_manager.go:414] "containerMap: removing container" podUID="891fe98c-6da1-4bd9-9d00-768831ba982f" containerName="payload-processing"
Oct 30 21:57:14.286438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286439    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891fe98c-6da1-4bd9-9d00-768831ba982f" containerName="payload-processing"
Oct 30 21:57:14.286667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286465    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3414edb7-7dae-493b-b3d2-884dc5ae2329" containerName="payload-processing"
Oct 30 21:57:14.286667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286473    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3414edb7-7dae-493b-b3d2-884dc5ae2329" containerName="payload-processing"
Oct 30 21:57:14.286667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286492    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" containerName="revoke-keys"
Oct 30 21:57:14.286667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286500    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" containerName="revoke-keys"
Oct 30 21:57:14.286880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286712    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="891fe98c-6da1-4bd9-9d00-768831ba982f" containerName="payload-processing"
Oct 30 21:57:14.286880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286729    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3414edb7-7dae-493b-b3d2-884dc5ae2329" containerName="payload-processing"
Oct 30 21:57:14.286880 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.286746    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" containerName="revoke-keys"
Oct 30 21:57:14.292224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.292203    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.318947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.318917    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:14.318947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.318926    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:14.319169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.319051    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:57:14.319723 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.319705    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:14.354246 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.354208    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"]
Oct 30 21:57:14.383662 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.383622    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.383876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.383744    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77vc\" (UniqueName: \"kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.384094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.384063    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.485802 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.485762    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.486003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.485828    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.486003 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.485874    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77vc\" (UniqueName: \"kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.486522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.486497    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.488262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.488241    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.505588 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.505556    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77vc\" (UniqueName: \"kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc\") pod \"maas-api-revoke-sub-b0d76819-84h5p\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.607783 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.607701    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:14.983460 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:14.983407    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"]
Oct 30 21:57:14.985431 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:14.985398    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a18cc_9f25_4f0f_bf0a_e39f55bd29d9.slice/crio-d6afba9343d43e80a553c0120f78e2c497c052654b23ac49f207278b71926c1b WatchSource:0}: Error finding container d6afba9343d43e80a553c0120f78e2c497c052654b23ac49f207278b71926c1b: Status 404 returned error can't find the container with id d6afba9343d43e80a553c0120f78e2c497c052654b23ac49f207278b71926c1b
Oct 30 21:57:15.261569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:15.261476    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p" event={"ID":"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9","Type":"ContainerStarted","Data":"d6afba9343d43e80a553c0120f78e2c497c052654b23ac49f207278b71926c1b"}
Oct 30 21:57:16.271472 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:16.271437    6695 generic.go:356] "Generic (PLEG): container finished" podID="208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" containerID="8b1c1b4d00aa79404bf047e8a54dff023d4fc8dbc115666481465390fe89d643" exitCode=0
Oct 30 21:57:16.271923 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:16.271514    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p" event={"ID":"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9","Type":"ContainerDied","Data":"8b1c1b4d00aa79404bf047e8a54dff023d4fc8dbc115666481465390fe89d643"}
Oct 30 21:57:17.632300 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.632265    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:17.752209 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.752169    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc\" (UniqueName: \"kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc\") pod \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") "
Oct 30 21:57:17.752408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.752387    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca\") pod \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") "
Oct 30 21:57:17.752471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.752429    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca\") pod \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\" (UID: \"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9\") "
Oct 30 21:57:17.752820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.752787    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca" pod "208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" (UID: "208a18cc-9f25-4f0f-bf0a-e39f55bd29d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:17.753262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.753239    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:17.754635 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.754592    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca" pod "208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" (UID: "208a18cc-9f25-4f0f-bf0a-e39f55bd29d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:17.754775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.754753    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc" pod "208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" (UID: "208a18cc-9f25-4f0f-bf0a-e39f55bd29d9"). InnerVolumeSpecName "kube-api-access-s77vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:17.855174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.855090    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s77vc\" (UniqueName: \"kubernetes.io/projected/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-kube-api-access-s77vc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:17.855174 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:17.855123    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:18.304528 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:18.304488    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p" event={"ID":"208a18cc-9f25-4f0f-bf0a-e39f55bd29d9","Type":"ContainerDied","Data":"d6afba9343d43e80a553c0120f78e2c497c052654b23ac49f207278b71926c1b"}
Oct 30 21:57:18.304731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:18.304559    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6afba9343d43e80a553c0120f78e2c497c052654b23ac49f207278b71926c1b"
Oct 30 21:57:18.304731 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:18.304505    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"
Oct 30 21:57:19.031848 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:19.031809    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"]
Oct 30 21:57:19.042486 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:19.042451    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb79667-b5wgj"]
Oct 30 21:57:19.216215 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:19.216181    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d534e28-ec0d-42fa-976e-af7e63d20caf" path="/var/lib/kubelet/pods/5d534e28-ec0d-42fa-976e-af7e63d20caf/volumes"
Oct 30 21:57:26.039422 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.039377    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"]
Oct 30 21:57:26.040307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.040286    6695 cpu_manager.go:414] "containerMap: removing container" podUID="208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" containerName="revoke-keys"
Oct 30 21:57:26.040353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.040308    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" containerName="revoke-keys"
Oct 30 21:57:26.040482 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.040473    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" containerName="revoke-keys"
Oct 30 21:57:26.046114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.046090    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.063261 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.063231    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"]
Oct 30 21:57:26.069000 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.068969    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 21:57:26.069178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.069020    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-055c7144178d\""
Oct 30 21:57:26.069178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.069075    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-xtj5d\""
Oct 30 21:57:26.069178 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.069166    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 21:57:26.071351 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.071329    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 21:57:26.071484 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.071359    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 21:57:26.187772 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.187727    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.187967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.187834    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.188020 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.187982    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.188066 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.188043    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bhv\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.289898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.289804    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.289898 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.289885    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bhv\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.290212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.289954    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.290212 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.290012    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.290523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.290495    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.290774 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.290752    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.292451 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.292432    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.308906 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.308880    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bhv\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.360693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.360652    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:26.682708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.682676    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"]
Oct 30 21:57:26.683881 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:26.683853    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b948544_9143_4c37_8b9a_dcf87d76aaf1.slice/crio-e8895d67152126b438e4c0c211fe280ebe69332dc545f068681143a7fb048d9b WatchSource:0}: Error finding container e8895d67152126b438e4c0c211fe280ebe69332dc545f068681143a7fb048d9b: Status 404 returned error can't find the container with id e8895d67152126b438e4c0c211fe280ebe69332dc545f068681143a7fb048d9b
Oct 30 21:57:26.747477 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:26.747441    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"]
Oct 30 21:57:27.393404 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:27.393364    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76" event={"ID":"6b948544-9143-4c37-8b9a-dcf87d76aaf1","Type":"ContainerStarted","Data":"2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e"}
Oct 30 21:57:27.393888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:27.393432    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76" event={"ID":"6b948544-9143-4c37-8b9a-dcf87d76aaf1","Type":"ContainerStarted","Data":"e8895d67152126b438e4c0c211fe280ebe69332dc545f068681143a7fb048d9b"}
Oct 30 21:57:27.393888 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:27.393467    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:27.435656 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:27.435581    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76" podStartSLOduration=1.435564552 podStartE2EDuration="1.435564552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:27.432586836 +0000 UTC m=+2830.355045143" watchObservedRunningTime="2026-10-30 21:57:27.435564552 +0000 UTC m=+2830.358022853"
Oct 30 21:57:29.037438 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:29.037401    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"]
Oct 30 21:57:29.048686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:29.048653    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2717da8-vkkxf"]
Oct 30 21:57:29.230100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:29.230060    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98d5d9f-e412-44f3-8432-22bd319d351b" path="/var/lib/kubelet/pods/b98d5d9f-e412-44f3-8432-22bd319d351b/volumes"
Oct 30 21:57:30.680967 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:30.680924    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"]
Oct 30 21:57:31.114712 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:31.114667    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:57:32.394297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.394246    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76" podUID="6b948544-9143-4c37-8b9a-dcf87d76aaf1" containerName="payload-processing" containerID="cri-o://2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e" gracePeriod=25
Oct 30 21:57:32.763522 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.763488    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:32.844633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.844580    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume\") pod \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") "
Oct 30 21:57:32.845062 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.845034    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume\") pod \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") "
Oct 30 21:57:32.845282 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.845261    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv\") pod \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") "
Oct 30 21:57:32.845713 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.845681    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials\") pod \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\" (UID: \"6b948544-9143-4c37-8b9a-dcf87d76aaf1\") "
Oct 30 21:57:32.847722 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.847431    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume" pod "6b948544-9143-4c37-8b9a-dcf87d76aaf1" (UID: "6b948544-9143-4c37-8b9a-dcf87d76aaf1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:32.848086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.848064    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume" pod "6b948544-9143-4c37-8b9a-dcf87d76aaf1" (UID: "6b948544-9143-4c37-8b9a-dcf87d76aaf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:32.858063 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.858013    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv" pod "6b948544-9143-4c37-8b9a-dcf87d76aaf1" (UID: "6b948544-9143-4c37-8b9a-dcf87d76aaf1"). InnerVolumeSpecName "kube-api-access-w6bhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:32.860076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.860034    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials" pod "6b948544-9143-4c37-8b9a-dcf87d76aaf1" (UID: "6b948544-9143-4c37-8b9a-dcf87d76aaf1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:32.948571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.948525    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.948571 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.948570    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bhv\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-kube-api-access-w6bhv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.948845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.948616    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b948544-9143-4c37-8b9a-dcf87d76aaf1-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.948845 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:32.948632    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b948544-9143-4c37-8b9a-dcf87d76aaf1-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:33.494503 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.494415    6695 generic.go:356] "Generic (PLEG): container finished" podID="6b948544-9143-4c37-8b9a-dcf87d76aaf1" containerID="2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e" exitCode=0
Oct 30 21:57:33.495169 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.494682    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76" event={"ID":"6b948544-9143-4c37-8b9a-dcf87d76aaf1","Type":"ContainerDied","Data":"2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e"}
Oct 30 21:57:33.495355 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.495335    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76" event={"ID":"6b948544-9143-4c37-8b9a-dcf87d76aaf1","Type":"ContainerDied","Data":"e8895d67152126b438e4c0c211fe280ebe69332dc545f068681143a7fb048d9b"}
Oct 30 21:57:33.495499 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.494761    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"
Oct 30 21:57:33.495724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.495371    6695 scope.go:122] "RemoveContainer" containerID="2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e"
Oct 30 21:57:33.520628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.515120    6695 scope.go:122] "RemoveContainer" containerID="2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e"
Oct 30 21:57:33.520628 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:33.515510    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e\": container with ID starting with 2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e not found: ID does not exist" containerID="2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e"
Oct 30 21:57:33.520628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.515550    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e"} err="failed to get container status \"2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e\": rpc error: code = NotFound desc = could not find container \"2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e\": container with ID starting with 2551b9fd24651f52ccaca10d84d510d9d7e7b7bc4c4fca7f6a5c1b063e0b954e not found: ID does not exist"
Oct 30 21:57:33.596701 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.596651    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"]
Oct 30 21:57:33.604928 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:33.604899    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645b6lj76"]
Oct 30 21:57:34.303626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.298760    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"]
Oct 30 21:57:34.303626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.300139    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6b948544-9143-4c37-8b9a-dcf87d76aaf1" containerName="payload-processing"
Oct 30 21:57:34.303626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.300165    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b948544-9143-4c37-8b9a-dcf87d76aaf1" containerName="payload-processing"
Oct 30 21:57:34.303626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.300528    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b948544-9143-4c37-8b9a-dcf87d76aaf1" containerName="payload-processing"
Oct 30 21:57:34.493661 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.493616    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.495657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.493497    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"]
Oct 30 21:57:34.508987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.508211    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:34.508987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.508329    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:57:34.508987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.508407    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:34.508987 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.508471    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:34.596740 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.596588    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.597043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.597002    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.597352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.597332    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggbz\" (UniqueName: \"kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.699638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.699566    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.699916 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.699893    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.700136 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.700120    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggbz\" (UniqueName: \"kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.701118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.701094    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.702874 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.702835    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.748694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.737286    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggbz\" (UniqueName: \"kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz\") pod \"maas-api-revoke-sub-f869ca42-6bqm7\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:34.929692 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:34.918765    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:35.227805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:35.227767    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b948544-9143-4c37-8b9a-dcf87d76aaf1" path="/var/lib/kubelet/pods/6b948544-9143-4c37-8b9a-dcf87d76aaf1/volumes"
Oct 30 21:57:35.432453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:35.432423    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"]
Oct 30 21:57:35.433829 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:35.433798    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eb3d60_953b_4af5_9420_8870674004b2.slice/crio-1d9e63e7614da14243a6b867ec866303a1c0bab246240cfed7f8506ec7a852d1 WatchSource:0}: Error finding container 1d9e63e7614da14243a6b867ec866303a1c0bab246240cfed7f8506ec7a852d1: Status 404 returned error can't find the container with id 1d9e63e7614da14243a6b867ec866303a1c0bab246240cfed7f8506ec7a852d1
Oct 30 21:57:35.614206 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:35.614167    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7" event={"ID":"c1eb3d60-953b-4af5-9420-8870674004b2","Type":"ContainerStarted","Data":"1d9e63e7614da14243a6b867ec866303a1c0bab246240cfed7f8506ec7a852d1"}
Oct 30 21:57:35.682274 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:35.682226    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285" podUID="6586ff36-960d-4541-a273-96223d5446a5" containerName="payload-processing" containerID="cri-o://fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076" gracePeriod=25
Oct 30 21:57:35.979575 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:35.979544    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:57:36.124937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.124826    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume\") pod \"6586ff36-960d-4541-a273-96223d5446a5\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") "
Oct 30 21:57:36.124937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.124888    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p\") pod \"6586ff36-960d-4541-a273-96223d5446a5\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") "
Oct 30 21:57:36.124937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.124921    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume\") pod \"6586ff36-960d-4541-a273-96223d5446a5\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") "
Oct 30 21:57:36.125267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.125036    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials\") pod \"6586ff36-960d-4541-a273-96223d5446a5\" (UID: \"6586ff36-960d-4541-a273-96223d5446a5\") "
Oct 30 21:57:36.125267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.125244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume" pod "6586ff36-960d-4541-a273-96223d5446a5" (UID: "6586ff36-960d-4541-a273-96223d5446a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:36.125453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.125278    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume" pod "6586ff36-960d-4541-a273-96223d5446a5" (UID: "6586ff36-960d-4541-a273-96223d5446a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:36.125685 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.125666    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:36.125765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.125688    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6586ff36-960d-4541-a273-96223d5446a5-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:36.127255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.127228    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p" pod "6586ff36-960d-4541-a273-96223d5446a5" (UID: "6586ff36-960d-4541-a273-96223d5446a5"). InnerVolumeSpecName "kube-api-access-r496p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:36.127357 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.127262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials" pod "6586ff36-960d-4541-a273-96223d5446a5" (UID: "6586ff36-960d-4541-a273-96223d5446a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:36.227033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.226998    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r496p\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-kube-api-access-r496p\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:36.227033 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.227027    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6586ff36-960d-4541-a273-96223d5446a5-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:36.624027 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.623992    6695 generic.go:356] "Generic (PLEG): container finished" podID="6586ff36-960d-4541-a273-96223d5446a5" containerID="fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076" exitCode=0
Oct 30 21:57:36.624498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.624042    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"
Oct 30 21:57:36.624498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.624072    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285" event={"ID":"6586ff36-960d-4541-a273-96223d5446a5","Type":"ContainerDied","Data":"fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076"}
Oct 30 21:57:36.624498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.624123    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-686b96ccfc-wk285" event={"ID":"6586ff36-960d-4541-a273-96223d5446a5","Type":"ContainerDied","Data":"5a334e0bdd514f096464d4e44e3700fdcc80d3d682f634db4c9cb86c87546652"}
Oct 30 21:57:36.624498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.624146    6695 scope.go:122] "RemoveContainer" containerID="fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076"
Oct 30 21:57:36.626202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.626175    6695 generic.go:356] "Generic (PLEG): container finished" podID="c1eb3d60-953b-4af5-9420-8870674004b2" containerID="a47ca6f4a1d4edf835e2e54df5768c2b31c23fc6085cf98ee20f91c660749082" exitCode=0
Oct 30 21:57:36.626330 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.626257    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7" event={"ID":"c1eb3d60-953b-4af5-9420-8870674004b2","Type":"ContainerDied","Data":"a47ca6f4a1d4edf835e2e54df5768c2b31c23fc6085cf98ee20f91c660749082"}
Oct 30 21:57:36.635569 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.635549    6695 scope.go:122] "RemoveContainer" containerID="fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076"
Oct 30 21:57:36.635892 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:36.635871    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076\": container with ID starting with fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076 not found: ID does not exist" containerID="fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076"
Oct 30 21:57:36.635979 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.635900    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076"} err="failed to get container status \"fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076\": rpc error: code = NotFound desc = could not find container \"fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076\": container with ID starting with fc98f0cc0b028fc507af3520bd3a495ec816d509a1f47e59086856fa5b56c076 not found: ID does not exist"
Oct 30 21:57:36.699485 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.699386    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"]
Oct 30 21:57:36.703728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:36.703697    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-686b96ccfc-wk285"]
Oct 30 21:57:37.215985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.215953    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6586ff36-960d-4541-a273-96223d5446a5" path="/var/lib/kubelet/pods/6586ff36-960d-4541-a273-96223d5446a5/volumes"
Oct 30 21:57:37.864106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.864078    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:37.951523 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.951419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz\" (UniqueName: \"kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz\") pod \"c1eb3d60-953b-4af5-9420-8870674004b2\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") "
Oct 30 21:57:37.951725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.951520    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca\") pod \"c1eb3d60-953b-4af5-9420-8870674004b2\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") "
Oct 30 21:57:37.951812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.951753    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca\") pod \"c1eb3d60-953b-4af5-9420-8870674004b2\" (UID: \"c1eb3d60-953b-4af5-9420-8870674004b2\") "
Oct 30 21:57:37.952024 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.951993    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca" pod "c1eb3d60-953b-4af5-9420-8870674004b2" (UID: "c1eb3d60-953b-4af5-9420-8870674004b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:37.952491 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.952466    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:37.954013 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.953975    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz" pod "c1eb3d60-953b-4af5-9420-8870674004b2" (UID: "c1eb3d60-953b-4af5-9420-8870674004b2"). InnerVolumeSpecName "kube-api-access-2ggbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:37.954372 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:37.954346    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca" pod "c1eb3d60-953b-4af5-9420-8870674004b2" (UID: "c1eb3d60-953b-4af5-9420-8870674004b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:38.054510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:38.054460    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1eb3d60-953b-4af5-9420-8870674004b2-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:38.054510 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:38.054504    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ggbz\" (UniqueName: \"kubernetes.io/projected/c1eb3d60-953b-4af5-9420-8870674004b2-kube-api-access-2ggbz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:38.645905 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:38.645872    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"
Oct 30 21:57:38.646107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:38.645899    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7" event={"ID":"c1eb3d60-953b-4af5-9420-8870674004b2","Type":"ContainerDied","Data":"1d9e63e7614da14243a6b867ec866303a1c0bab246240cfed7f8506ec7a852d1"}
Oct 30 21:57:38.646107 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:38.645951    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d9e63e7614da14243a6b867ec866303a1c0bab246240cfed7f8506ec7a852d1"
Oct 30 21:57:39.781100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.781061    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"]
Oct 30 21:57:39.782017 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.781994    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6586ff36-960d-4541-a273-96223d5446a5" containerName="payload-processing"
Oct 30 21:57:39.782100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.782019    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6586ff36-960d-4541-a273-96223d5446a5" containerName="payload-processing"
Oct 30 21:57:39.782100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.782056    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c1eb3d60-953b-4af5-9420-8870674004b2" containerName="revoke-keys"
Oct 30 21:57:39.782100 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.782062    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1eb3d60-953b-4af5-9420-8870674004b2" containerName="revoke-keys"
Oct 30 21:57:39.782267 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.782254    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6586ff36-960d-4541-a273-96223d5446a5" containerName="payload-processing"
Oct 30 21:57:39.782307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.782278    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1eb3d60-953b-4af5-9420-8870674004b2" containerName="revoke-keys"
Oct 30 21:57:39.786454 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.786428    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.835568 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.835529    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"]
Oct 30 21:57:39.844973 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.844941    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:57:39.845386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.845063    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:39.845386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.845130    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:39.845386 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.845177    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:39.877779 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.877740    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscdn\" (UniqueName: \"kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.877995 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.877964    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.878144 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.878123    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.979920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.979877    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscdn\" (UniqueName: \"kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.980106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.980009    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.980208 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.980109    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.980758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.980732    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.982758 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.982731    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:39.999896 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:39.999864    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscdn\" (UniqueName: \"kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn\") pod \"maas-api-revoke-sub-e6f2d897-2p27f\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:40.104392 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:40.104309    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:40.416667 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:40.416643    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"]
Oct 30 21:57:40.420645 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:40.420081    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3000f88e_b5f3_4b52_a2b4_d75063792c4b.slice/crio-bba7a84380043d7132c6abd645586cfa6ae586d8adb9686aa9e806c2bf18af2f WatchSource:0}: Error finding container bba7a84380043d7132c6abd645586cfa6ae586d8adb9686aa9e806c2bf18af2f: Status 404 returned error can't find the container with id bba7a84380043d7132c6abd645586cfa6ae586d8adb9686aa9e806c2bf18af2f
Oct 30 21:57:40.679544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:40.679460    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f" event={"ID":"3000f88e-b5f3-4b52-a2b4-d75063792c4b","Type":"ContainerStarted","Data":"bba7a84380043d7132c6abd645586cfa6ae586d8adb9686aa9e806c2bf18af2f"}
Oct 30 21:57:41.689324 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:41.689234    6695 generic.go:356] "Generic (PLEG): container finished" podID="3000f88e-b5f3-4b52-a2b4-d75063792c4b" containerID="f6c85f791feaedc4905c32a00834bb4fdc645878223b60aa378c989e07fbfc55" exitCode=0
Oct 30 21:57:41.689724 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:41.689326    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f" event={"ID":"3000f88e-b5f3-4b52-a2b4-d75063792c4b","Type":"ContainerDied","Data":"f6c85f791feaedc4905c32a00834bb4fdc645878223b60aa378c989e07fbfc55"}
Oct 30 21:57:42.030947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.030910    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"]
Oct 30 21:57:42.035366 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.035339    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6802ba-97xcl"]
Oct 30 21:57:42.890707 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.890681    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:42.927594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.927559    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca\") pod \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") "
Oct 30 21:57:42.927782 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.927723    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn\" (UniqueName: \"kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn\") pod \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") "
Oct 30 21:57:42.928005 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.927981    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca" pod "3000f88e-b5f3-4b52-a2b4-d75063792c4b" (UID: "3000f88e-b5f3-4b52-a2b4-d75063792c4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:42.928086 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.928061    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca\") pod \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\" (UID: \"3000f88e-b5f3-4b52-a2b4-d75063792c4b\") "
Oct 30 21:57:42.928867 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.928819    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:42.934947 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.934915    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca" pod "3000f88e-b5f3-4b52-a2b4-d75063792c4b" (UID: "3000f88e-b5f3-4b52-a2b4-d75063792c4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:42.935121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:42.935102    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn" pod "3000f88e-b5f3-4b52-a2b4-d75063792c4b" (UID: "3000f88e-b5f3-4b52-a2b4-d75063792c4b"). InnerVolumeSpecName "kube-api-access-rscdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:43.030546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:43.030509    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rscdn\" (UniqueName: \"kubernetes.io/projected/3000f88e-b5f3-4b52-a2b4-d75063792c4b-kube-api-access-rscdn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:43.030546 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:43.030541    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3000f88e-b5f3-4b52-a2b4-d75063792c4b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:43.216749 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:43.216711    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ad1c09-940d-4adc-b003-ef91008bf5ac" path="/var/lib/kubelet/pods/b6ad1c09-940d-4adc-b003-ef91008bf5ac/volumes"
Oct 30 21:57:43.709314 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:43.709283    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"
Oct 30 21:57:43.709502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:43.709277    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f" event={"ID":"3000f88e-b5f3-4b52-a2b4-d75063792c4b","Type":"ContainerDied","Data":"bba7a84380043d7132c6abd645586cfa6ae586d8adb9686aa9e806c2bf18af2f"}
Oct 30 21:57:43.709502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:43.709421    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bba7a84380043d7132c6abd645586cfa6ae586d8adb9686aa9e806c2bf18af2f"
Oct 30 21:57:44.932385 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.932347    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"]
Oct 30 21:57:44.933278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.933260    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3000f88e-b5f3-4b52-a2b4-d75063792c4b" containerName="revoke-keys"
Oct 30 21:57:44.933325 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.933279    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3000f88e-b5f3-4b52-a2b4-d75063792c4b" containerName="revoke-keys"
Oct 30 21:57:44.933408 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.933398    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3000f88e-b5f3-4b52-a2b4-d75063792c4b" containerName="revoke-keys"
Oct 30 21:57:44.937742 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.937720    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:44.956550 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.956521    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"]
Oct 30 21:57:44.962367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.962330    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:44.962367 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.962335    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:44.962544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.962392    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:57:44.962820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:44.962803    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:45.062061 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.062020    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.062289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.062265    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9n97\" (UniqueName: \"kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.062452 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.062435    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.164058 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.164017    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9n97\" (UniqueName: \"kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.164276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.164170    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.164276 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.164269    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.165301 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.165269    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.166988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.166964    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.183395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.183330    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9n97\" (UniqueName: \"kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97\") pod \"maas-api-revoke-sub-fcf4799f-8sbcr\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.253317 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.253279    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:45.568885 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.568851    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"]
Oct 30 21:57:45.571093 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:45.571062    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b17a2c_eb3e_4d0b_bbac_7e845ff4960f.slice/crio-fcf6207d0878f343970340ccc25d53dec4098aa76a3ec1f641c329499c18eefe WatchSource:0}: Error finding container fcf6207d0878f343970340ccc25d53dec4098aa76a3ec1f641c329499c18eefe: Status 404 returned error can't find the container with id fcf6207d0878f343970340ccc25d53dec4098aa76a3ec1f641c329499c18eefe
Oct 30 21:57:45.728570 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:45.728525    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr" event={"ID":"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f","Type":"ContainerStarted","Data":"fcf6207d0878f343970340ccc25d53dec4098aa76a3ec1f641c329499c18eefe"}
Oct 30 21:57:46.169305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.169223    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"]
Oct 30 21:57:46.174205 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.174179    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.197052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.197015    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"]
Oct 30 21:57:46.284548 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.284499    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.284743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.284640    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.284743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.284692    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.284921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.284852    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.284921 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.284910    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.285039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.284982    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.285161 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.285145    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.285233 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.285195    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.285305 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.285285    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386502 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386515    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386554    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386593    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386686 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386665    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386726    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386820    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386846    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.386935 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.386885    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.387189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.387077    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.387189 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.387116    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.387306 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.387205    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.387402 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.387380    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.387665 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.387648    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.389050 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.389026    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.389303 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.389287    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.405383 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.405358    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.405543 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.405523    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b\") pod \"maas-default-gateway-openshift-default-77f556d647-j6jq9\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.515164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.515132    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:46.740475 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.740439    6695 generic.go:356] "Generic (PLEG): container finished" podID="79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" containerID="a9a00650ff61d36a9f804a53866c347915f02329a12adc325d6a952e3f476720" exitCode=0
Oct 30 21:57:46.740743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.740514    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr" event={"ID":"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f","Type":"ContainerDied","Data":"a9a00650ff61d36a9f804a53866c347915f02329a12adc325d6a952e3f476720"}
Oct 30 21:57:46.849114 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.849083    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"]
Oct 30 21:57:46.851501 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:46.851470    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94169247_0d0d_4d28_aa55_039baffc64ed.slice/crio-13d52b3d53ea48c17817358f30bce246ba096c8590a2db4ae876393dd6fc872c WatchSource:0}: Error finding container 13d52b3d53ea48c17817358f30bce246ba096c8590a2db4ae876393dd6fc872c: Status 404 returned error can't find the container with id 13d52b3d53ea48c17817358f30bce246ba096c8590a2db4ae876393dd6fc872c
Oct 30 21:57:46.854580 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.854545    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:46.854671 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.854656    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:46.854726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:46.854693    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:47.751278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:47.751238    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" event={"ID":"94169247-0d0d-4d28-aa55-039baffc64ed","Type":"ContainerStarted","Data":"cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099"}
Oct 30 21:57:47.751767 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:47.751300    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" event={"ID":"94169247-0d0d-4d28-aa55-039baffc64ed","Type":"ContainerStarted","Data":"13d52b3d53ea48c17817358f30bce246ba096c8590a2db4ae876393dd6fc872c"}
Oct 30 21:57:47.841795 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:47.831917    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podStartSLOduration=1.831896155 podStartE2EDuration="1.831896155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:47.828469244 +0000 UTC m=+2850.750927555" watchObservedRunningTime="2026-10-30 21:57:47.831896155 +0000 UTC m=+2850.754354455"
Oct 30 21:57:47.983816 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:47.983788    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:48.149911 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.149874    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca\") pod \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") "
Oct 30 21:57:48.150139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.149979    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca\") pod \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") "
Oct 30 21:57:48.150139 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.150070    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97\" (UniqueName: \"kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97\") pod \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\" (UID: \"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f\") "
Oct 30 21:57:48.150313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.150290    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca" pod "79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" (UID: "79b17a2c-eb3e-4d0b-bbac-7e845ff4960f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:48.150689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.150659    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:48.152219 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.152193    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca" pod "79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" (UID: "79b17a2c-eb3e-4d0b-bbac-7e845ff4960f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:48.152316 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.152260    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97" pod "79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" (UID: "79b17a2c-eb3e-4d0b-bbac-7e845ff4960f"). InnerVolumeSpecName "kube-api-access-n9n97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:48.253281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.253241    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9n97\" (UniqueName: \"kubernetes.io/projected/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-kube-api-access-n9n97\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:48.253281 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.253268    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:48.534808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.531011    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:48.537155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.537117    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:48.537313 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.537187    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:48.861962 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.861861    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr" event={"ID":"79b17a2c-eb3e-4d0b-bbac-7e845ff4960f","Type":"ContainerDied","Data":"fcf6207d0878f343970340ccc25d53dec4098aa76a3ec1f641c329499c18eefe"}
Oct 30 21:57:48.862646 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.862623    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcf6207d0878f343970340ccc25d53dec4098aa76a3ec1f641c329499c18eefe"
Oct 30 21:57:48.862846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:48.862760    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"
Oct 30 21:57:49.536082 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:49.536046    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:49.536297 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:49.536104    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:50.536654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:50.535767    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:50.536654 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:50.535856    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:51.521628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:51.515684    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:51.521628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:51.515749    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:51.607594 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:51.607535    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"]
Oct 30 21:57:52.537628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:52.537111    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:52.537628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:52.537174    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:53.516170 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.516136    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:53.516708 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.516674    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:53.807202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.807158    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"]
Oct 30 21:57:53.808448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.808422    6695 cpu_manager.go:414] "containerMap: removing container" podUID="79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" containerName="revoke-keys"
Oct 30 21:57:53.808542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.808451    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" containerName="revoke-keys"
Oct 30 21:57:53.808728 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.808713    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" containerName="revoke-keys"
Oct 30 21:57:53.814202 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.814171    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:53.832344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.832304    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"]
Oct 30 21:57:53.837625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.835191    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 21:57:53.837625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.835916    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-fhcdg\""
Oct 30 21:57:53.837625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.835978    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-3adc28cfce40\""
Oct 30 21:57:53.837625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.836035    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 21:57:53.837625 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.836107    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 21:57:53.844623 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.842859    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 21:57:53.988636 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.981619    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:53.996634 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.988964    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:53.996983 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:53.996947    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfvz\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.051043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.051001    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.156327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.156168    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.156518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.156328    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.156518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.156378    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.156518 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.156413    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfvz\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.160672 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.160638    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.166593 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.166564    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.170622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.170230    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.182622 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.180714    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfvz\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.437797 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.437697    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:54.517388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.516951    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:54.517388 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.517010    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:54.779818 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:54.779779    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"]
Oct 30 21:57:54.782657 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:57:54.782593    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e81593_4356_4296_9736_9dae8c77fb77.slice/crio-10b959f6c2e34f6a3302b5c9015951214b96312cb4c45c8fa53e3d63db63c0c4 WatchSource:0}: Error finding container 10b959f6c2e34f6a3302b5c9015951214b96312cb4c45c8fa53e3d63db63c0c4: Status 404 returned error can't find the container with id 10b959f6c2e34f6a3302b5c9015951214b96312cb4c45c8fa53e3d63db63c0c4
Oct 30 21:57:55.120224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:55.116959    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc" event={"ID":"51e81593-4356-4296-9736-9dae8c77fb77","Type":"ContainerStarted","Data":"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9"}
Oct 30 21:57:55.120224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:55.117026    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc" event={"ID":"51e81593-4356-4296-9736-9dae8c77fb77","Type":"ContainerStarted","Data":"10b959f6c2e34f6a3302b5c9015951214b96312cb4c45c8fa53e3d63db63c0c4"}
Oct 30 21:57:55.120224 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:55.117960    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:57:55.159395 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:55.159334    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc" podStartSLOduration=2.159318623 podStartE2EDuration="2.159318623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:55.155572779 +0000 UTC m=+2858.078031079" watchObservedRunningTime="2026-10-30 21:57:55.159318623 +0000 UTC m=+2858.081776920"
Oct 30 21:57:55.515999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:55.515914    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:57:55.515999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:55.515971    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused"
Oct 30 21:57:56.608619 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:56.608571    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt" podUID="0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" containerName="payload-processing" containerID="cri-o://8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c" gracePeriod=25
Oct 30 21:57:56.679999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:56.679868    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:56.681456 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:56.681369    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:56.682250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:56.682223    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 21:57:57.007183 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.007152    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:57:57.115193 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.115110    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"]
Oct 30 21:57:57.115985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.115969    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" containerName="payload-processing"
Oct 30 21:57:57.116049 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.115986    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" containerName="payload-processing"
Oct 30 21:57:57.116121 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.116112    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" containerName="payload-processing"
Oct 30 21:57:57.120948 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.120918    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.131700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.131497    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume\") pod \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") "
Oct 30 21:57:57.131700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.131619    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials\") pod \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") "
Oct 30 21:57:57.131700 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.131681    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume\") pod \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") "
Oct 30 21:57:57.132026 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.131817    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r\") pod \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\" (UID: \"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42\") "
Oct 30 21:57:57.133985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.132164    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume" pod "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" (UID: "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:57.133985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.132775    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume" pod "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" (UID: "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:57.133985 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.132812    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.135539 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.135509    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials" pod "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" (UID: "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:57.138541 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.138512    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r" pod "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" (UID: "0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42"). InnerVolumeSpecName "kube-api-access-bl52r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:57.146529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.146500    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"]
Oct 30 21:57:57.149693 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.149662    6695 generic.go:356] "Generic (PLEG): container finished" podID="0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" containerID="8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c" exitCode=0
Oct 30 21:57:57.149826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.149745    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"
Oct 30 21:57:57.149826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.149746    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt" event={"ID":"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42","Type":"ContainerDied","Data":"8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c"}
Oct 30 21:57:57.149826 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.149800    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt" event={"ID":"0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42","Type":"ContainerDied","Data":"790a23e27abfa507dcdadd5f9b484d22674651d3a0b5d9dee0f97a1bfbb79b6a"}
Oct 30 21:57:57.149964 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.149829    6695 scope.go:122] "RemoveContainer" containerID="8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c"
Oct 30 21:57:57.186508 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.186487    6695 scope.go:122] "RemoveContainer" containerID="8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c"
Oct 30 21:57:57.186838 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.186814    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c\": container with ID starting with 8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c not found: ID does not exist" containerID="8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c"
Oct 30 21:57:57.186900 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.186856    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c"} err="failed to get container status \"8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c\": rpc error: code = NotFound desc = could not find container \"8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c\": container with ID starting with 8b430f3387f1a1a252129ee767706b54b8cfafb148c8a8fbe704824eeafdc23c not found: ID does not exist"
Oct 30 21:57:57.220956 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.220921    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"]
Oct 30 21:57:57.226098 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.226070    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-6678nb7nt"]
Oct 30 21:57:57.234352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.234320    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kkn\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.234498 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.234368    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.234614 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.234566    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.235043 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.235015    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.235262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.235232    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.235262 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.235259    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.235396 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.235274    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl52r\" (UniqueName: \"kubernetes.io/projected/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42-kube-api-access-bl52r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:57.336691 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.336651    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.336876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.336738    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kkn\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.336876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.336763    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.336876 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.336807    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.337021 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.336979    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3d5d13/agc-cred-e899723f90d2: secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:57.337021 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.336998    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt: secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:57.337113 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.337074    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials podName:0d9d7956-498b-47fe-aaba-3ef3ba790f5e nodeName:}" failed. No retries permitted until 2026-10-30 21:57:57.837052536 +0000 UTC m=+2860.759510856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt" (UID: "0d9d7956-498b-47fe-aaba-3ef3ba790f5e") : secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:57.337413 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.337393    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.337453 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.337440    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.359056 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.359029    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kkn\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.656118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.656084    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"]
Oct 30 21:57:57.661134 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.661102    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"]
Oct 30 21:57:57.661564 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.661537    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt" podUID="0d9d7956-498b-47fe-aaba-3ef3ba790f5e"
Oct 30 21:57:57.854879 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:57.854835    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:57.855144 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.854951    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3d5d13/agc-cred-e899723f90d2: secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:57.855144 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.854973    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt: secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:57.855144 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:57.855043    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials podName:0d9d7956-498b-47fe-aaba-3ef3ba790f5e nodeName:}" failed. No retries permitted until 2026-10-30 21:57:58.855022695 +0000 UTC m=+2861.777480989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt" (UID: "0d9d7956-498b-47fe-aaba-3ef3ba790f5e") : secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:58.191289 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.191248    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:58.203074 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.203041    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:58.386537 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.386494    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume\") pod \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") "
Oct 30 21:57:58.386805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.386706    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn\") pod \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") "
Oct 30 21:57:58.386805 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.386772    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume\") pod \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") "
Oct 30 21:57:58.386937 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.386912    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume" pod "0d9d7956-498b-47fe-aaba-3ef3ba790f5e" (UID: "0d9d7956-498b-47fe-aaba-3ef3ba790f5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:58.387078 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.387057    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume" pod "0d9d7956-498b-47fe-aaba-3ef3ba790f5e" (UID: "0d9d7956-498b-47fe-aaba-3ef3ba790f5e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:58.387545 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.387525    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.387696 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.387547    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.388954 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.388935    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn" pod "0d9d7956-498b-47fe-aaba-3ef3ba790f5e" (UID: "0d9d7956-498b-47fe-aaba-3ef3ba790f5e"). InnerVolumeSpecName "kube-api-access-79kkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:58.489527 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.489441    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kkn\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-kube-api-access-79kkn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:57:58.894883 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:58.894838    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt\" (UID: \"0d9d7956-498b-47fe-aaba-3ef3ba790f5e\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:58.895429 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:58.895008    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3d5d13/agc-cred-e899723f90d2: secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:58.895429 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:58.895032    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt: secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:58.895429 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:57:58.895099    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials podName:0d9d7956-498b-47fe-aaba-3ef3ba790f5e nodeName:}" failed. No retries permitted until 2026-10-30 21:58:00.895078438 +0000 UTC m=+2863.817536727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt" (UID: "0d9d7956-498b-47fe-aaba-3ef3ba790f5e") : secret "agc-cred-e899723f90d2" not found
Oct 30 21:57:59.201028 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:59.200947    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"
Oct 30 21:57:59.216887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:59.216850    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42" path="/var/lib/kubelet/pods/0dfbb711-ffe2-4ffe-a6c5-7ff9b9f15c42/volumes"
Oct 30 21:57:59.278360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:59.278325    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"]
Oct 30 21:57:59.286084 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:59.286045    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-6f99p7vtt"]
Oct 30 21:57:59.400806 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:57:59.400770    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d9d7956-498b-47fe-aaba-3ef3ba790f5e-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:01.091310 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:58:01.091255    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:58:01.216288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:01.216245    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9d7956-498b-47fe-aaba-3ef3ba790f5e" path="/var/lib/kubelet/pods/0d9d7956-498b-47fe-aaba-3ef3ba790f5e/volumes"
Oct 30 21:58:02.656559 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.656507    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w" podUID="3014f8c4-83ea-41c2-b04b-417f6b69a764" containerName="payload-processing" containerID="cri-o://aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb" gracePeriod=25
Oct 30 21:58:02.954564 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.954533    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:58:02.973481 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.973446    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials\") pod \"3014f8c4-83ea-41c2-b04b-417f6b69a764\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") "
Oct 30 21:58:02.973737 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.973579    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume\") pod \"3014f8c4-83ea-41c2-b04b-417f6b69a764\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") "
Oct 30 21:58:02.973812 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.973735    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume\") pod \"3014f8c4-83ea-41c2-b04b-417f6b69a764\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") "
Oct 30 21:58:02.973877 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.973855    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw\") pod \"3014f8c4-83ea-41c2-b04b-417f6b69a764\" (UID: \"3014f8c4-83ea-41c2-b04b-417f6b69a764\") "
Oct 30 21:58:02.974094 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.974068    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume" pod "3014f8c4-83ea-41c2-b04b-417f6b69a764" (UID: "3014f8c4-83ea-41c2-b04b-417f6b69a764"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:02.974250 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.974066    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume" pod "3014f8c4-83ea-41c2-b04b-417f6b69a764" (UID: "3014f8c4-83ea-41c2-b04b-417f6b69a764"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:02.974490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.974470    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:02.974544 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.974494    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3014f8c4-83ea-41c2-b04b-417f6b69a764-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:02.976147 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.976122    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw" pod "3014f8c4-83ea-41c2-b04b-417f6b69a764" (UID: "3014f8c4-83ea-41c2-b04b-417f6b69a764"). InnerVolumeSpecName "kube-api-access-f6gfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:02.976253 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:02.976151    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials" pod "3014f8c4-83ea-41c2-b04b-417f6b69a764" (UID: "3014f8c4-83ea-41c2-b04b-417f6b69a764"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:03.075789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.075739    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gfw\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-kube-api-access-f6gfw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:03.075789 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.075785    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3014f8c4-83ea-41c2-b04b-417f6b69a764-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:03.240255 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.240214    6695 generic.go:356] "Generic (PLEG): container finished" podID="3014f8c4-83ea-41c2-b04b-417f6b69a764" containerID="aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb" exitCode=0
Oct 30 21:58:03.240471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.240268    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"
Oct 30 21:58:03.240471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.240285    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w" event={"ID":"3014f8c4-83ea-41c2-b04b-417f6b69a764","Type":"ContainerDied","Data":"aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb"}
Oct 30 21:58:03.240471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.240346    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w" event={"ID":"3014f8c4-83ea-41c2-b04b-417f6b69a764","Type":"ContainerDied","Data":"b397386b9faf0c543aba114ac24570a4e55a872c8d8283a818a2715c0b459835"}
Oct 30 21:58:03.240471 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.240376    6695 scope.go:122] "RemoveContainer" containerID="aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb"
Oct 30 21:58:03.252339 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.252311    6695 scope.go:122] "RemoveContainer" containerID="aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb"
Oct 30 21:58:03.252713 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:58:03.252685    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb\": container with ID starting with aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb not found: ID does not exist" containerID="aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb"
Oct 30 21:58:03.252846 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.252717    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb"} err="failed to get container status \"aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb\": rpc error: code = NotFound desc = could not find container \"aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb\": container with ID starting with aade62af7fe23fbb51fddd80788fd6b5005884b0c49d7538ab46efba9bccf5eb not found: ID does not exist"
Oct 30 21:58:03.293875 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.293836    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"]
Oct 30 21:58:03.304820 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:03.304783    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-d746h747w"]
Oct 30 21:58:05.261633 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:05.242533    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3014f8c4-83ea-41c2-b04b-417f6b69a764" path="/var/lib/kubelet/pods/3014f8c4-83ea-41c2-b04b-417f6b69a764/volumes"
Oct 30 21:58:07.155444 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.155394    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 21:58:07.382889 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.382842    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"]
Oct 30 21:58:07.384265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.384201    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3014f8c4-83ea-41c2-b04b-417f6b69a764" containerName="payload-processing"
Oct 30 21:58:07.384265 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.384264    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3014f8c4-83ea-41c2-b04b-417f6b69a764" containerName="payload-processing"
Oct 30 21:58:07.384495 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.384479    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3014f8c4-83ea-41c2-b04b-417f6b69a764" containerName="payload-processing"
Oct 30 21:58:07.391051 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.390992    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.410458    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8d814a2cf4e\""
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.410460    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"]
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.410527    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b35fff36b219\""
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.424826    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nrl\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.424889    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.424976    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.425461 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.425051    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.460887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.460258    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d9c68187ac0c\""
Oct 30 21:58:07.460887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.460382    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:58:07.460887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.460472    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:58:07.460887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.460519    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:58:07.460887 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.460674    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rxxkz\""
Oct 30 21:58:07.461675 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.461645    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:58:07.527097 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.527040    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.527295 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.527145    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.527463 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.527437    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nrl\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.527560 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.527482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.528831 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.528794    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.529201 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.529178    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.531430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.531357    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.564496 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.564428    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nrl\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl\") pod \"payload-processing-external-model-5bc9559f7b-dmc57\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") " pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:07.782041 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:07.781998    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:09.018307 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:09.018258    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"]
Oct 30 21:58:09.839629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:09.839157    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57" event={"ID":"ef922e2b-6068-4ae6-9015-26566381cd3b","Type":"ContainerStarted","Data":"6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0"}
Oct 30 21:58:09.839629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:09.839231    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57" event={"ID":"ef922e2b-6068-4ae6-9015-26566381cd3b","Type":"ContainerStarted","Data":"eb9a4366869cd10f31f02975b50e51c1b6e43479e34e533526a161ef653ff64a"}
Oct 30 21:58:09.843638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:09.840476    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:09.898542 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:09.898470    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57" podStartSLOduration=2.898446059 podStartE2EDuration="2.898446059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:09.883334312 +0000 UTC m=+2872.805792623" watchObservedRunningTime="2026-10-30 21:58:09.898446059 +0000 UTC m=+2872.820904361"
Oct 30 21:58:13.081187 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.081145    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"]
Oct 30 21:58:13.112363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.112327    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"]
Oct 30 21:58:13.117853 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.117829    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.140847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.140806    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bba74b814b2d\""
Oct 30 21:58:13.150694 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.150661    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"]
Oct 30 21:58:13.263657 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.263583    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpptn\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.263847 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.263771    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.263952 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.263935    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.264011 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.263965    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.365158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.365064    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.365158 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.365108    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.365430 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.365310    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpptn\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.365500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.365478    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.366106 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.366054    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.366240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.366121    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.368155 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.368131    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.383490 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.383466    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpptn\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn\") pod \"payload-processing-external-model-f6d747c94-sf29l\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") " pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.447419 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.447380    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:13.754342 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.754306    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"]
Oct 30 21:58:13.756509 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:58:13.756483    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a11982_bcde_40b4_9518_ccfab9564ae0.slice/crio-5949f4a9be504856afb01a135d9ccc5ec4c2df56a449a491d11e727fc54da505 WatchSource:0}: Error finding container 5949f4a9be504856afb01a135d9ccc5ec4c2df56a449a491d11e727fc54da505: Status 404 returned error can't find the container with id 5949f4a9be504856afb01a135d9ccc5ec4c2df56a449a491d11e727fc54da505
Oct 30 21:58:13.990893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.990692    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l" event={"ID":"f0a11982-bcde-40b4-9518-ccfab9564ae0","Type":"ContainerStarted","Data":"e9133c6e3875aefaa88ae50ed32b426319788e8711c4bb336443ca8a42ef209a"}
Oct 30 21:58:13.990893 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.990759    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l" event={"ID":"f0a11982-bcde-40b4-9518-ccfab9564ae0","Type":"ContainerStarted","Data":"5949f4a9be504856afb01a135d9ccc5ec4c2df56a449a491d11e727fc54da505"}
Oct 30 21:58:13.991869 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:13.991840    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:14.029994 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:14.029940    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l" podStartSLOduration=1.029927602 podStartE2EDuration="1.029927602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:14.029258518 +0000 UTC m=+2876.951716819" watchObservedRunningTime="2026-10-30 21:58:14.029927602 +0000 UTC m=+2876.952385900"
Oct 30 21:58:18.081500 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.081453    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57" podUID="ef922e2b-6068-4ae6-9015-26566381cd3b" containerName="payload-processing" containerID="cri-o://6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0" gracePeriod=25
Oct 30 21:58:18.379652 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.379621    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:18.549165 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.549129    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl\") pod \"ef922e2b-6068-4ae6-9015-26566381cd3b\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") "
Oct 30 21:58:18.549363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.549235    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials\") pod \"ef922e2b-6068-4ae6-9015-26566381cd3b\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") "
Oct 30 21:58:18.549363 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.549309    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume\") pod \"ef922e2b-6068-4ae6-9015-26566381cd3b\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") "
Oct 30 21:58:18.549512 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.549380    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume\") pod \"ef922e2b-6068-4ae6-9015-26566381cd3b\" (UID: \"ef922e2b-6068-4ae6-9015-26566381cd3b\") "
Oct 30 21:58:18.549743 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.549698    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume" pod "ef922e2b-6068-4ae6-9015-26566381cd3b" (UID: "ef922e2b-6068-4ae6-9015-26566381cd3b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:18.549860 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.549781    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume" pod "ef922e2b-6068-4ae6-9015-26566381cd3b" (UID: "ef922e2b-6068-4ae6-9015-26566381cd3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:18.550113 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.550092    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:18.550164 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.550120    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef922e2b-6068-4ae6-9015-26566381cd3b-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:18.551450 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.551428    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials" pod "ef922e2b-6068-4ae6-9015-26566381cd3b" (UID: "ef922e2b-6068-4ae6-9015-26566381cd3b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:18.551529 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.551464    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl" pod "ef922e2b-6068-4ae6-9015-26566381cd3b" (UID: "ef922e2b-6068-4ae6-9015-26566381cd3b"). InnerVolumeSpecName "kube-api-access-98nrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:18.651288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.651201    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nrl\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-kube-api-access-98nrl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:18.651288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:18.651232    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ef922e2b-6068-4ae6-9015-26566381cd3b-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:58:19.036411 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.036377    6695 generic.go:356] "Generic (PLEG): container finished" podID="ef922e2b-6068-4ae6-9015-26566381cd3b" containerID="6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0" exitCode=0
Oct 30 21:58:19.036649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.036425    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"
Oct 30 21:58:19.036649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.036458    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57" event={"ID":"ef922e2b-6068-4ae6-9015-26566381cd3b","Type":"ContainerDied","Data":"6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0"}
Oct 30 21:58:19.036649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.036511    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57" event={"ID":"ef922e2b-6068-4ae6-9015-26566381cd3b","Type":"ContainerDied","Data":"eb9a4366869cd10f31f02975b50e51c1b6e43479e34e533526a161ef653ff64a"}
Oct 30 21:58:19.036649 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.036536    6695 scope.go:122] "RemoveContainer" containerID="6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0"
Oct 30 21:58:19.054029 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.053997    6695 scope.go:122] "RemoveContainer" containerID="6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0"
Oct 30 21:58:19.054432 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:58:19.054397    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0\": container with ID starting with 6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0 not found: ID does not exist" containerID="6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0"
Oct 30 21:58:19.054561 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.054436    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0"} err="failed to get container status \"6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0\": rpc error: code = NotFound desc = could not find container \"6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0\": container with ID starting with 6d32fc58d03f99906279de34999ca68b23ed04adc96f07a19548137646bc69e0 not found: ID does not exist"
Oct 30 21:58:19.091039 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.091004    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"]
Oct 30 21:58:19.101727 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.101696    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc9559f7b-dmc57"]
Oct 30 21:58:19.216288 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:19.216254    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef922e2b-6068-4ae6-9015-26566381cd3b" path="/var/lib/kubelet/pods/ef922e2b-6068-4ae6-9015-26566381cd3b/volumes"
Oct 30 21:58:22.917988 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:22.917948    6695 scope.go:122] "RemoveContainer" containerID="9e50362acdac4a623f4eab31ae609392d396a8f2dd5babce7671a25a0a531868"
Oct 30 21:58:22.929725 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:22.929701    6695 scope.go:122] "RemoveContainer" containerID="280ac4606c32f01311c59115753d7a450acd1c3b1a9d02e24ac1600e7411f538"
Oct 30 21:58:24.029488 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:24.029440    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"]
Oct 30 21:58:24.040428 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:24.040397    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f432bbf-m9hlj"]
Oct 30 21:58:25.215790 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:25.215757    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bdee61-9987-408e-b85c-54375f857b17" path="/var/lib/kubelet/pods/f7bdee61-9987-408e-b85c-54375f857b17/volumes"
Oct 30 21:58:26.010638 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:26.010576    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 21:58:31.076125 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:58:31.076040    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:58:35.087715 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:35.087671    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"]
Oct 30 21:58:35.122628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:35.122190    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9c476b-8pjkk"]
Oct 30 21:58:35.247863 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:35.247695    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f97aec-fd57-404a-b15c-e024c11a6c78" path="/var/lib/kubelet/pods/b4f97aec-fd57-404a-b15c-e024c11a6c78/volumes"
Oct 30 21:58:38.645765 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:38.645714    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:38.848629 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:38.847766    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:38.910924 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:38.910787    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:58:38.992943 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:38.991728    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.111753 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:39.111698    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:58:41.897263 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.897216    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"]
Oct 30 21:58:41.898327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.898300    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ef922e2b-6068-4ae6-9015-26566381cd3b" containerName="payload-processing"
Oct 30 21:58:41.898327 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.898327    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef922e2b-6068-4ae6-9015-26566381cd3b" containerName="payload-processing"
Oct 30 21:58:41.898547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.898532    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef922e2b-6068-4ae6-9015-26566381cd3b" containerName="payload-processing"
Oct 30 21:58:41.904348 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.904324    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:41.920240 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.920202    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"routing-overlay\""
Oct 30 21:58:41.923060 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.923031    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"]
Oct 30 21:58:41.928557 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.928528    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"openshift-service-ca.crt\""
Oct 30 21:58:41.928775 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.928720    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-9ad7991aeaf0\""
Oct 30 21:58:41.929283 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.928937    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-e2e--f95d90a5-dockercfg-25h7b\""
Oct 30 21:58:41.960256 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.960203    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:41.960457 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.960362    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:41.960515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.960490    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:41.960800 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:41.960777    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2ll\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.020443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.020409    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"kube-root-ca.crt\""
Oct 30 21:58:42.020443 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.020409    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-plugins-e2e-extmod-b-846656\""
Oct 30 21:58:42.075628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.069136    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.075628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.069374    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2ll\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.075628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.069482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.075628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.069546    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.075628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.070586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.075628 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.071245    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.145111 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.145036    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.206578 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.206467    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2ll\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll\") pod \"payload-processing-external-model-e2e-extmod-b-846656-55588v6t6\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.219920 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.219882    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:42.564547 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:42.564513    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"]
Oct 30 21:58:42.566172 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:58:42.566141    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd615063_3ebf_4b00_ace6_0ee6f5a0076a.slice/crio-9cb70cb0b26f84537aa4a49aaef9e7efd50c7ccd28a2a0b1668c8036a009dd91 WatchSource:0}: Error finding container 9cb70cb0b26f84537aa4a49aaef9e7efd50c7ccd28a2a0b1668c8036a009dd91: Status 404 returned error can't find the container with id 9cb70cb0b26f84537aa4a49aaef9e7efd50c7ccd28a2a0b1668c8036a009dd91
Oct 30 21:58:43.438866 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:43.438829    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6" event={"ID":"dd615063-3ebf-4b00-ace6-0ee6f5a0076a","Type":"ContainerStarted","Data":"3970c90ced84f73a05ee819ffc5b4252ac995a965d082ef5e659f18bd409ddaf"}
Oct 30 21:58:43.439353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:43.438893    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6" event={"ID":"dd615063-3ebf-4b00-ace6-0ee6f5a0076a","Type":"ContainerStarted","Data":"9cb70cb0b26f84537aa4a49aaef9e7efd50c7ccd28a2a0b1668c8036a009dd91"}
Oct 30 21:58:43.439353 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:43.439069    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:58:43.482171 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:43.482116    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6" podStartSLOduration=2.482101721 podStartE2EDuration="2.482101721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:43.477499133 +0000 UTC m=+2906.399957430" watchObservedRunningTime="2026-10-30 21:58:43.482101721 +0000 UTC m=+2906.404560056"
Oct 30 21:58:48.035773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:48.035732    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"]
Oct 30 21:58:48.046763 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:48.046724    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f595711d-v854n"]
Oct 30 21:58:49.215840 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:49.215806    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5571c25c-dbf5-45e5-911f-18610ca5469d" path="/var/lib/kubelet/pods/5571c25c-dbf5-45e5-911f-18610ca5469d/volumes"
Oct 30 21:58:54.450830 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:58:54.450798    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 21:59:01.075520 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:59:01.075483    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:59:22.034626 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:22.028913    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"]
Oct 30 21:59:22.042035 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:22.041999    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d99fe8a9-v69z9"]
Oct 30 21:59:22.998515 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:22.998475    6695 scope.go:122] "RemoveContainer" containerID="7703d30c1cac5a70283bee38aad814c55dc4b7773a480cebf5dc6be35d8cf755"
Oct 30 21:59:23.011168 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:23.011147    6695 scope.go:122] "RemoveContainer" containerID="e0fa2e056f2530d6257625629362760ed7c54370c629e8cb6cfc4cea408f8cd0"
Oct 30 21:59:23.022045 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:23.022026    6695 scope.go:122] "RemoveContainer" containerID="165f0192abafab122d2b9a4431ac05a8bd1017e33cb71c03ee93d02c0324bdb0"
Oct 30 21:59:23.216756 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:23.216726    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408e97d1-7502-47b6-9c05-8bdb948338e2" path="/var/lib/kubelet/pods/408e97d1-7502-47b6-9c05-8bdb948338e2/volumes"
Oct 30 21:59:31.074843 ip-10-0-142-232 kubenswrapper[6695]: E1030 21:59:31.074803    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 21:59:33.029096 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:33.029058    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"]
Oct 30 21:59:33.040235 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:33.040202    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0254599e-j56ht"]
Oct 30 21:59:33.216969 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:33.216936    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41e3d3f-cbe0-49df-9794-a7db9bffd145" path="/var/lib/kubelet/pods/a41e3d3f-cbe0-49df-9794-a7db9bffd145/volumes"
Oct 30 21:59:45.032010 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:45.031973    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"]
Oct 30 21:59:45.042081 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:45.042052    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e13ac79-tg9bs"]
Oct 30 21:59:45.216442 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:45.216406    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be863741-1fc7-4af3-b7d4-a288f0f82731" path="/var/lib/kubelet/pods/be863741-1fc7-4af3-b7d4-a288f0f82731/volumes"
Oct 30 21:59:46.033371 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:46.033312    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"]
Oct 30 21:59:46.044052 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:46.044020    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744700f5-7sdzk"]
Oct 30 21:59:47.302076 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:47.302029    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27bb615f-cf21-4fa6-bc03-d7d96f422447" path="/var/lib/kubelet/pods/27bb615f-cf21-4fa6-bc03-d7d96f422447/volumes"
Oct 30 21:59:51.030315 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.030277    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"]
Oct 30 21:59:51.035397 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.035369    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.053352 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.053319    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"]
Oct 30 21:59:51.059645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.059589    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:51.059645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.059628    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:59:51.059931 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.059674    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:59:51.060364 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.060343    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:59:51.135531 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.135497    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.135754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.135545    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbpj\" (UniqueName: \"kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.135754 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.135701    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.237726 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.237687    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.237942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.237738    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbpj\" (UniqueName: \"kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.237942 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.237809    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.238448 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.238421    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.240278 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.240258    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.258101 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.258072    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbpj\" (UniqueName: \"kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj\") pod \"maas-api-revoke-sub-74142843-n27s5\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.351360 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.351264    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:51.659808 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:51.659785    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"]
Oct 30 21:59:51.661822 ip-10-0-142-232 kubenswrapper[6695]: W1030 21:59:51.661795    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7ffce9_2236_4940_b5ee_dd4376ddf8cb.slice/crio-9b9d87d30abe28e045ad05e5ed10e38fbbe607e4be187ff8b882042410bb7091 WatchSource:0}: Error finding container 9b9d87d30abe28e045ad05e5ed10e38fbbe607e4be187ff8b882042410bb7091: Status 404 returned error can't find the container with id 9b9d87d30abe28e045ad05e5ed10e38fbbe607e4be187ff8b882042410bb7091
Oct 30 21:59:52.348699 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:52.348660    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5" event={"ID":"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb","Type":"ContainerStarted","Data":"e04880d14c71a0ef54e6373b1320011d7ed5dc1faedaac601403e148b9af83c4"}
Oct 30 21:59:52.349118 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:52.348720    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5" event={"ID":"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb","Type":"ContainerStarted","Data":"9b9d87d30abe28e045ad05e5ed10e38fbbe607e4be187ff8b882042410bb7091"}
Oct 30 21:59:52.413891 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:52.413836    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5" podStartSLOduration=1.4138157439999999 podStartE2EDuration="1.413815744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:52.403170858 +0000 UTC m=+2975.325629157" watchObservedRunningTime="2026-10-30 21:59:52.413815744 +0000 UTC m=+2975.336274049"
Oct 30 21:59:53.358903 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:53.358865    6695 generic.go:356] "Generic (PLEG): container finished" podID="cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" containerID="e04880d14c71a0ef54e6373b1320011d7ed5dc1faedaac601403e148b9af83c4" exitCode=0
Oct 30 21:59:53.359439 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:53.358949    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5" event={"ID":"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb","Type":"ContainerDied","Data":"e04880d14c71a0ef54e6373b1320011d7ed5dc1faedaac601403e148b9af83c4"}
Oct 30 21:59:54.545176 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.545146    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:54.593470 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.593435    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj\" (UniqueName: \"kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj\") pod \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") "
Oct 30 21:59:54.593689 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.593574    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca\") pod \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") "
Oct 30 21:59:54.593773 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.593736    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca\") pod \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\" (UID: \"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb\") "
Oct 30 21:59:54.593999 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.593976    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca" pod "cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" (UID: "cb7ffce9-2236-4940-b5ee-dd4376ddf8cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:54.594589 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.594568    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:59:54.596015 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.595963    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca" pod "cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" (UID: "cb7ffce9-2236-4940-b5ee-dd4376ddf8cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:54.596142 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.596084    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj" pod "cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" (UID: "cb7ffce9-2236-4940-b5ee-dd4376ddf8cb"). InnerVolumeSpecName "kube-api-access-nqbpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:54.696105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.696013    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqbpj\" (UniqueName: \"kubernetes.io/projected/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-kube-api-access-nqbpj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:59:54.696105 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:54.696050    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 21:59:55.380344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:55.380299    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"
Oct 30 21:59:55.380344 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:55.380331    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5" event={"ID":"cb7ffce9-2236-4940-b5ee-dd4376ddf8cb","Type":"ContainerDied","Data":"9b9d87d30abe28e045ad05e5ed10e38fbbe607e4be187ff8b882042410bb7091"}
Oct 30 21:59:55.380651 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:55.380401    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9d87d30abe28e045ad05e5ed10e38fbbe607e4be187ff8b882042410bb7091"
Oct 30 21:59:57.034551 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:57.034507    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"]
Oct 30 21:59:57.044423 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:57.044385    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fabe4-xx2wm"]
Oct 30 21:59:57.215738 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:57.215703    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1763a0-fc71-4e23-889b-0649db2465ea" path="/var/lib/kubelet/pods/7c1763a0-fc71-4e23-889b-0649db2465ea/volumes"
Oct 30 21:59:58.027517 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:58.027479    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"]
Oct 30 21:59:58.038645 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:58.038598    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d47766f-kwfms"]
Oct 30 21:59:59.215658 ip-10-0-142-232 kubenswrapper[6695]: I1030 21:59:59.215551    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa21fc2-dcc1-4a5f-a628-e630065b872f" path="/var/lib/kubelet/pods/bfa21fc2-dcc1-4a5f-a628-e630065b872f/volumes"
Oct 30 22:00:00.145977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.145944    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"]
Oct 30 22:00:00.147190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.147167    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" containerName="revoke-keys"
Oct 30 22:00:00.147329 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.147191    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" containerName="revoke-keys"
Oct 30 22:00:00.147412 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.147399    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" containerName="revoke-keys"
Oct 30 22:00:00.153370 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.153343    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"]
Oct 30 22:00:00.153507 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.153474    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.158300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.158277    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"]
Oct 30 22:00:00.158441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.158391    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.163410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.163386    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"]
Oct 30 22:00:00.163572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.163558    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.169075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.169056    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"]
Oct 30 22:00:00.169218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.169174    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.173867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.173842    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:00.173987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.173872    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:00:00.173987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.173932    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:00.174449 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.174432    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.179368 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.179346    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"]
Oct 30 22:00:00.194765 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.194730    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"]
Oct 30 22:00:00.195117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.195088    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.202496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.202464    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"]
Oct 30 22:00:00.218460 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.218180    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"]
Oct 30 22:00:00.225944 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.225908    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"]
Oct 30 22:00:00.228084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.228060    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"]
Oct 30 22:00:00.240821 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.240768    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"]
Oct 30 22:00:00.268055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.268018    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"]
Oct 30 22:00:00.275626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.268492    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.297652 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.291676    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"]
Oct 30 22:00:00.301277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.301210    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.302873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.302757    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfz7\" (UniqueName: \"kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7\") pod \"maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.302873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.302805    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.303121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.302900    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r799l\" (UniqueName: \"kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l\") pod \"maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.303121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.302928    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.303121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.302963    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzxz\" (UniqueName: \"kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.303121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.302991    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2ls\" (UniqueName: \"kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls\") pod \"maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.303121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.303022    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzjs\" (UniqueName: \"kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs\") pod \"maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.303121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.303092    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9h4\" (UniqueName: \"kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4\") pod \"maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.303414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.303156    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.303414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.303208    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.303414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.303236    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.303414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.303366    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.332902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.332728    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"]
Oct 30 22:00:00.332902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.332794    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.340070 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.340047    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"]
Oct 30 22:00:00.340228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.340094    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"]
Oct 30 22:00:00.341829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.341323    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.346710 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.346690    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"]
Oct 30 22:00:00.347138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.347107    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.351879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.351858    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"]
Oct 30 22:00:00.351997 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.351981    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.357501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.356866    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"]
Oct 30 22:00:00.357501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.356915    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"]
Oct 30 22:00:00.357501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.356944    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"]
Oct 30 22:00:00.358549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.358528    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.363971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.363951    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"]
Oct 30 22:00:00.364052 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.363992    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"]
Oct 30 22:00:00.364103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.364063    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.365538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.365520    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"]
Oct 30 22:00:00.369356 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.369337    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"]
Oct 30 22:00:00.378082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.378057    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"]
Oct 30 22:00:00.382709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.382685    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"]
Oct 30 22:00:00.388909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.388890    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"]
Oct 30 22:00:00.389046 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.389028    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.395782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.395758    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.395879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.395861    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"]
Oct 30 22:00:00.401283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.401234    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"]
Oct 30 22:00:00.405314 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405289    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"]
Oct 30 22:00:00.405688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405668    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.405789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405702    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzxz\" (UniqueName: \"kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.405789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405724    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2ls\" (UniqueName: \"kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls\") pod \"maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.405789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405753    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzjs\" (UniqueName: \"kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs\") pod \"maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.406031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405797    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.406031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405837    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.406031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405931    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9h4\" (UniqueName: \"kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4\") pod \"maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.406031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.405990    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.406031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406027    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.406361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406059    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.406361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406137    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngmfq\" (UniqueName: \"kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.406361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406235    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.406361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406272    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9bd\" (UniqueName: \"kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd\") pod \"maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.406361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406304    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.406668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406396    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzgk\" (UniqueName: \"kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk\") pod \"maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.406668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406505    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.406668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406505    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfz7\" (UniqueName: \"kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7\") pod \"maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.406668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406568    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.406900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406719    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r799l\" (UniqueName: \"kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l\") pod \"maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.406900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406816    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.406900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406833    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.406900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.406853    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.407342 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.407324    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.407485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.407464    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.426042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.426009    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzjs\" (UniqueName: \"kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs\") pod \"maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.426195 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.426122    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9h4\" (UniqueName: \"kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4\") pod \"maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.426736 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.426708    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2ls\" (UniqueName: \"kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls\") pod \"maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.427245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.427221    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzxz\" (UniqueName: \"kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.427392 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.427368    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r799l\" (UniqueName: \"kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l\") pod \"maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.427444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.427402    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfz7\" (UniqueName: \"kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7\") pod \"maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.473789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.473761    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:00.496934 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.496906    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:00.508451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508419    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgtz\" (UniqueName: \"kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.508663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508502    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.508663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508570    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.508663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508647    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.508891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508695    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69b24\" (UniqueName: \"kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24\") pod \"maas-api-key-cleanup-29889960-tqlnx\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.508891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508779    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztkn\" (UniqueName: \"kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.508891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508846    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6g6n\" (UniqueName: \"kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.509125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508947    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bk5\" (UniqueName: \"kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5\") pod \"maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.509125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.508997    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngmfq\" (UniqueName: \"kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.509125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509035    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.509125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509067    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509145    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptxg\" (UniqueName: \"kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509173    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9bd\" (UniqueName: \"kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd\") pod \"maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509196    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509225    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509271    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-tqlnx\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509287    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509297    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.509366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509303    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.509825 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509566    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzgk\" (UniqueName: \"kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk\") pod \"maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.509825 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509689    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khd2r\" (UniqueName: \"kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.509825 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509726    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.509825 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.509805    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.520207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.520180    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:00.530956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.530873    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngmfq\" (UniqueName: \"kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.533543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.533515    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzgk\" (UniqueName: \"kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk\") pod \"maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.536491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.536456    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9bd\" (UniqueName: \"kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd\") pod \"maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.584352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.583842    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:00.606708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.606671    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.649832    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.649997    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652195    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652239    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69b24\" (UniqueName: \"kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24\") pod \"maas-api-key-cleanup-29889960-tqlnx\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652289    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztkn\" (UniqueName: \"kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652359    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6g6n\" (UniqueName: \"kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652408    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bk5\" (UniqueName: \"kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5\") pod \"maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652449    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652477    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652543    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptxg\" (UniqueName: \"kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652587    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652673    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-tqlnx\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652705    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652835    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khd2r\" (UniqueName: \"kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652878    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.652972    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgtz\" (UniqueName: \"kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.657503    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.695971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.691861    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:00.721231 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.721194    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.722975 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.722941    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.726229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.723529    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-tqlnx\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.726229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.724418    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.726229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.725225    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.726229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.725829    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.730296 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.729877    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69b24\" (UniqueName: \"kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24\") pod \"maas-api-key-cleanup-29889960-tqlnx\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.739938 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.739909    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6g6n\" (UniqueName: \"kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.740619 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.740582    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:00.740866 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.740844    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:00.748156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.743339    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khd2r\" (UniqueName: \"kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:00.771039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.771012    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:00.776952 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.776886    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptxg\" (UniqueName: \"kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:00.781326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.781255    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgtz\" (UniqueName: \"kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.782039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.782014    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bk5\" (UniqueName: \"kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5\") pod \"maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.784686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.784660    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:00.796659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.789371    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztkn\" (UniqueName: \"kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:00.989226 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.986162    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:00.996365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:00.995933    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:01.006262 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.005732    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:01.025057 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.023947    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:01.032942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.032583    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"]
Oct 30 22:00:01.041742 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.041634    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"]
Oct 30 22:00:01.141235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.141205    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"]
Oct 30 22:00:01.150078 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:01.149549    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e1398e_f2fa_4b74_8525_a7e2017f2fcc.slice/crio-03dc0e2f3b108f3837e10b55982789762af780724168611cb077f6b683f89bc7 WatchSource:0}: Error finding container 03dc0e2f3b108f3837e10b55982789762af780724168611cb077f6b683f89bc7: Status 404 returned error can't find the container with id 03dc0e2f3b108f3837e10b55982789762af780724168611cb077f6b683f89bc7
Oct 30 22:00:01.150685 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:01.150645    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:00:01.264840 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:01.264794    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377405e1_8481_444f_a61a_5e59dcf38460.slice/crio-d86fbec8fc07c37036c9402650d86034dc62ba2c2aebf8af97b578950fc6f073 WatchSource:0}: Error finding container d86fbec8fc07c37036c9402650d86034dc62ba2c2aebf8af97b578950fc6f073: Status 404 returned error can't find the container with id d86fbec8fc07c37036c9402650d86034dc62ba2c2aebf8af97b578950fc6f073
Oct 30 22:00:01.267404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.267333    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"]
Oct 30 22:00:01.308445 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.308276    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"]
Oct 30 22:00:01.313532 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:01.311868    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded66b735_a7c1_470c_a775_11657620f643.slice/crio-6f4225e002c5a451d2f5f97ede57a7081eca6f622a3e2b8484aaa7975977065b WatchSource:0}: Error finding container 6f4225e002c5a451d2f5f97ede57a7081eca6f622a3e2b8484aaa7975977065b: Status 404 returned error can't find the container with id 6f4225e002c5a451d2f5f97ede57a7081eca6f622a3e2b8484aaa7975977065b
Oct 30 22:00:01.344138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.344088    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"]
Oct 30 22:00:01.419961 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.419931    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"]
Oct 30 22:00:01.447364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.447333    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"]
Oct 30 22:00:01.451550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.451515    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" event={"ID":"377405e1-8481-444f-a61a-5e59dcf38460","Type":"ContainerStarted","Data":"d86fbec8fc07c37036c9402650d86034dc62ba2c2aebf8af97b578950fc6f073"}
Oct 30 22:00:01.454196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.454167    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" event={"ID":"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc","Type":"ContainerStarted","Data":"03dc0e2f3b108f3837e10b55982789762af780724168611cb077f6b683f89bc7"}
Oct 30 22:00:01.456322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.455859    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerStarted","Data":"5f9c53b248d68b872bbe805668ddf53b855ad63d2bfd135b7dbdce1c924416fe"}
Oct 30 22:00:01.457895 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.457872    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" event={"ID":"ed66b735-a7c1-470c-a775-11657620f643","Type":"ContainerStarted","Data":"6f4225e002c5a451d2f5f97ede57a7081eca6f622a3e2b8484aaa7975977065b"}
Oct 30 22:00:01.460767 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.460729    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerStarted","Data":"7de5c13dfa1510aad7c8b9f3f40fe04ead96fc072c0cc2d24497329312802b35"}
Oct 30 22:00:01.488180 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.488147    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"]
Oct 30 22:00:01.499375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.499344    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"]
Oct 30 22:00:01.510633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.507998    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"]
Oct 30 22:00:01.546171 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.546101    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"]
Oct 30 22:00:01.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.551950    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"]
Oct 30 22:00:01.556661 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.556633    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"]
Oct 30 22:00:01.564553 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.564526    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"]
Oct 30 22:00:01.625154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:01.625126    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"]
Oct 30 22:00:01.719745 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:01.719705    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2542d6_ffac_4d16_9cb0_f6f9ab97e51f.slice/crio-5443fb9c7cdeb89f6aa20c281d8e59aeda2a4ddce3ddb35122a60638667fb3c0 WatchSource:0}: Error finding container 5443fb9c7cdeb89f6aa20c281d8e59aeda2a4ddce3ddb35122a60638667fb3c0: Status 404 returned error can't find the container with id 5443fb9c7cdeb89f6aa20c281d8e59aeda2a4ddce3ddb35122a60638667fb3c0
Oct 30 22:00:02.474480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.474444    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" event={"ID":"c94a3b92-eb35-437d-8edb-5683bef4f768","Type":"ContainerStarted","Data":"80bca83916b5239c151bebb276257cccd87d5c901ee649b45357cbd4c169e909"}
Oct 30 22:00:02.475877 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.475849    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" event={"ID":"2a505c48-0f05-449a-bce4-570c9e6f82cd","Type":"ContainerStarted","Data":"af2cf397aefbdf1d70de7d0f72db1de09184bf6b47fe1ea604df3cf8743f6c1d"}
Oct 30 22:00:02.477224 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.477199    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerStarted","Data":"5443fb9c7cdeb89f6aa20c281d8e59aeda2a4ddce3ddb35122a60638667fb3c0"}
Oct 30 22:00:02.478981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.478957    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" event={"ID":"d381869f-da79-449f-8a22-1970a5ef7dfa","Type":"ContainerStarted","Data":"698d50a9b7f88adad586dbb8c1a7a841d41a84fcdd8db7d0cdf3d78732c4f592"}
Oct 30 22:00:02.479100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.479004    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" event={"ID":"d381869f-da79-449f-8a22-1970a5ef7dfa","Type":"ContainerStarted","Data":"1c73cfd415a0361e9416735a8e21ec47f03a6311085f62612d7b4d05f0aee1e1"}
Oct 30 22:00:02.481113 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.481086    6695 generic.go:356] "Generic (PLEG): container finished" podID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerID="2cc1f6837ced31cf3b15bddca5a9d405b4b1c6e104fe2c3b5509ada77faf5af5" exitCode=60
Oct 30 22:00:02.481206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.481162    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerDied","Data":"2cc1f6837ced31cf3b15bddca5a9d405b4b1c6e104fe2c3b5509ada77faf5af5"}
Oct 30 22:00:02.481493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.481472    6695 scope.go:122] "RemoveContainer" containerID="2cc1f6837ced31cf3b15bddca5a9d405b4b1c6e104fe2c3b5509ada77faf5af5"
Oct 30 22:00:02.482756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.482733    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" event={"ID":"3fa2035c-7a77-4bc7-9141-c912cf623256","Type":"ContainerStarted","Data":"21da69969c3762218800957538ece66adf6e7f0de0d260fc8de80b6015ba4af3"}
Oct 30 22:00:02.483871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.483845    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"]
Oct 30 22:00:02.484187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.484167    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" event={"ID":"a9928370-f1a1-4126-9df2-376c65322ed3","Type":"ContainerStarted","Data":"f004f9b79a58c5e2267ab1f6929f2adbb90a7d5f44384e8bba1a9bc496e9a33e"}
Oct 30 22:00:02.485659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.485549    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" event={"ID":"f0df621f-cede-41ef-b8d0-b895f48034d5","Type":"ContainerStarted","Data":"18a6e7d0392a8fbd4cb420cef6a7e28f302d6ccfd5b08194dfe737a9266d498f"}
Oct 30 22:00:02.487269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.487247    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerStarted","Data":"fd2b7a4edf8bbd3cad9b4ca06bce009714731072251b55f951f5d02377c65d09"}
Oct 30 22:00:02.488650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.488580    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" event={"ID":"8193d2c6-b8f1-4460-adc6-5eefdda788d9","Type":"ContainerStarted","Data":"129c4e27b3a8fb3ec1f7a3f88e5a586de6519265f9aa95dc4f2fff47ee48cc6c"}
Oct 30 22:00:02.489941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.489920    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerStarted","Data":"8dc16c909c47e6a679f61e1e113daf43f52de69084bd11697feea749466a2426"}
Oct 30 22:00:02.491513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.491491    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" event={"ID":"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc","Type":"ContainerStarted","Data":"2ba3af77a2c5249efb1ff22d9318a3eeabf779ab97e0cce1276f90a8f27146f6"}
Oct 30 22:00:02.499016 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.498988    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerStarted","Data":"b6e07179c0012cf5b6d11cbf807e0af52ca10c1ead3d520ecb739eb7e7cc9563"}
Oct 30 22:00:02.500429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.500406    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" event={"ID":"4ed668a4-c548-4e09-8d1d-09a08c500c37","Type":"ContainerStarted","Data":"66c6360050922e774109c0931ac580a65d183fe74e206886cfafcc632bc9e03b"}
Oct 30 22:00:02.522918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.522855    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" podStartSLOduration=2.522834149 podStartE2EDuration="2.522834149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.516741796 +0000 UTC m=+2985.439200099" watchObservedRunningTime="2026-10-30 22:00:02.522834149 +0000 UTC m=+2985.445292448"
Oct 30 22:00:02.559360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.559270    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" podStartSLOduration=2.559249906 podStartE2EDuration="2.559249906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.552647538 +0000 UTC m=+2985.475105838" watchObservedRunningTime="2026-10-30 22:00:02.559249906 +0000 UTC m=+2985.481708205"
Oct 30 22:00:02.666758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:02.666698    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" podStartSLOduration=2.666682384 podStartE2EDuration="2.666682384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.663430905 +0000 UTC m=+2985.585889204" watchObservedRunningTime="2026-10-30 22:00:02.666682384 +0000 UTC m=+2985.589140682"
Oct 30 22:00:03.143417 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.143378    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"]
Oct 30 22:00:03.149945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.149149    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.165883 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.165829    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:03.166612 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.166574    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"]
Oct 30 22:00:03.174546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.174514    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.175178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.174859    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.175178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.174967    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttl6r\" (UniqueName: \"kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.214416 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:03.214371    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd381869f_da79_449f_8a22_1970a5ef7dfa.slice/crio-698d50a9b7f88adad586dbb8c1a7a841d41a84fcdd8db7d0cdf3d78732c4f592.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df3ce61_1578_490a_b58b_3342bc5b1a01.slice/crio-b6e07179c0012cf5b6d11cbf807e0af52ca10c1ead3d520ecb739eb7e7cc9563.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377405e1_8481_444f_a61a_5e59dcf38460.slice/crio-conmon-214a620ea26e616a74c2d21ffcbc2a1c4ec09e1ee1ad8a4390fe3854c92cd0c9.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:03.276217 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.276178    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.277168 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.277136    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.277401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.277221    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.277401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.277297    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttl6r\" (UniqueName: \"kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.313421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.313018    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.314029 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.313807    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttl6r\" (UniqueName: \"kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r\") pod \"maas-api-revoke-sub-1de3e169-x6cpr\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.513571 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.513531    6695 generic.go:356] "Generic (PLEG): container finished" podID="377405e1-8481-444f-a61a-5e59dcf38460" containerID="214a620ea26e616a74c2d21ffcbc2a1c4ec09e1ee1ad8a4390fe3854c92cd0c9" exitCode=60
Oct 30 22:00:03.514040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.513638    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" event={"ID":"377405e1-8481-444f-a61a-5e59dcf38460","Type":"ContainerDied","Data":"214a620ea26e616a74c2d21ffcbc2a1c4ec09e1ee1ad8a4390fe3854c92cd0c9"}
Oct 30 22:00:03.514137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.514018    6695 scope.go:122] "RemoveContainer" containerID="214a620ea26e616a74c2d21ffcbc2a1c4ec09e1ee1ad8a4390fe3854c92cd0c9"
Oct 30 22:00:03.515944 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.515911    6695 generic.go:356] "Generic (PLEG): container finished" podID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerID="2c01c6ee4652ed073cbaebcf32d1c7dee4156846c11be67b1023ea3a94644918" exitCode=60
Oct 30 22:00:03.516063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.515977    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerDied","Data":"2c01c6ee4652ed073cbaebcf32d1c7dee4156846c11be67b1023ea3a94644918"}
Oct 30 22:00:03.516312 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.516297    6695 scope.go:122] "RemoveContainer" containerID="2c01c6ee4652ed073cbaebcf32d1c7dee4156846c11be67b1023ea3a94644918"
Oct 30 22:00:03.520156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.520133    6695 generic.go:356] "Generic (PLEG): container finished" podID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerID="2ba3af77a2c5249efb1ff22d9318a3eeabf779ab97e0cce1276f90a8f27146f6" exitCode=60
Oct 30 22:00:03.520259 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.520209    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" event={"ID":"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc","Type":"ContainerDied","Data":"2ba3af77a2c5249efb1ff22d9318a3eeabf779ab97e0cce1276f90a8f27146f6"}
Oct 30 22:00:03.520504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.520487    6695 scope.go:122] "RemoveContainer" containerID="2ba3af77a2c5249efb1ff22d9318a3eeabf779ab97e0cce1276f90a8f27146f6"
Oct 30 22:00:03.521886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.521866    6695 generic.go:356] "Generic (PLEG): container finished" podID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerID="b6e07179c0012cf5b6d11cbf807e0af52ca10c1ead3d520ecb739eb7e7cc9563" exitCode=60
Oct 30 22:00:03.521976 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.521949    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerDied","Data":"b6e07179c0012cf5b6d11cbf807e0af52ca10c1ead3d520ecb739eb7e7cc9563"}
Oct 30 22:00:03.522181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.522161    6695 scope.go:122] "RemoveContainer" containerID="b6e07179c0012cf5b6d11cbf807e0af52ca10c1ead3d520ecb739eb7e7cc9563"
Oct 30 22:00:03.523701 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.523678    6695 generic.go:356] "Generic (PLEG): container finished" podID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerID="13b393f644172d4a9e77464e74cf06281095e82beb7634280ba02a5ccc04ef0a" exitCode=60
Oct 30 22:00:03.523801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.523758    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" event={"ID":"4ed668a4-c548-4e09-8d1d-09a08c500c37","Type":"ContainerDied","Data":"13b393f644172d4a9e77464e74cf06281095e82beb7634280ba02a5ccc04ef0a"}
Oct 30 22:00:03.523947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.523933    6695 scope.go:122] "RemoveContainer" containerID="13b393f644172d4a9e77464e74cf06281095e82beb7634280ba02a5ccc04ef0a"
Oct 30 22:00:03.525421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.525399    6695 generic.go:356] "Generic (PLEG): container finished" podID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerID="91fbd565980c8ac00346871cb737f384a54e7a4f55ff5ad78ebf0e26c585a9d3" exitCode=60
Oct 30 22:00:03.525494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.525476    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" event={"ID":"c94a3b92-eb35-437d-8edb-5683bef4f768","Type":"ContainerDied","Data":"91fbd565980c8ac00346871cb737f384a54e7a4f55ff5ad78ebf0e26c585a9d3"}
Oct 30 22:00:03.525678 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.525664    6695 scope.go:122] "RemoveContainer" containerID="91fbd565980c8ac00346871cb737f384a54e7a4f55ff5ad78ebf0e26c585a9d3"
Oct 30 22:00:03.527261 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.527239    6695 generic.go:356] "Generic (PLEG): container finished" podID="ed66b735-a7c1-470c-a775-11657620f643" containerID="23e458c9044ce683925fd6055a9903270f96aae2b1c6345589541ca24e110dec" exitCode=60
Oct 30 22:00:03.527362 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.527326    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" event={"ID":"ed66b735-a7c1-470c-a775-11657620f643","Type":"ContainerDied","Data":"23e458c9044ce683925fd6055a9903270f96aae2b1c6345589541ca24e110dec"}
Oct 30 22:00:03.527524 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.527507    6695 scope.go:122] "RemoveContainer" containerID="23e458c9044ce683925fd6055a9903270f96aae2b1c6345589541ca24e110dec"
Oct 30 22:00:03.529968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.529949    6695 generic.go:356] "Generic (PLEG): container finished" podID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerID="9e4735254e6315a6c06706b16aa731c919f0b570d25761a34264d0235c83488e" exitCode=60
Oct 30 22:00:03.530201 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.530177    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" event={"ID":"2a505c48-0f05-449a-bce4-570c9e6f82cd","Type":"ContainerDied","Data":"9e4735254e6315a6c06706b16aa731c919f0b570d25761a34264d0235c83488e"}
Oct 30 22:00:03.530333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.530321    6695 scope.go:122] "RemoveContainer" containerID="9e4735254e6315a6c06706b16aa731c919f0b570d25761a34264d0235c83488e"
Oct 30 22:00:03.531855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.531820    6695 generic.go:356] "Generic (PLEG): container finished" podID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerID="da1169f288bbf7ca47429d98c70b5664c9ccc055cd687353ca4e9784e30773bf" exitCode=60
Oct 30 22:00:03.531951 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.531901    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerDied","Data":"da1169f288bbf7ca47429d98c70b5664c9ccc055cd687353ca4e9784e30773bf"}
Oct 30 22:00:03.532066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.532048    6695 scope.go:122] "RemoveContainer" containerID="da1169f288bbf7ca47429d98c70b5664c9ccc055cd687353ca4e9784e30773bf"
Oct 30 22:00:03.533798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.533767    6695 generic.go:356] "Generic (PLEG): container finished" podID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerID="698d50a9b7f88adad586dbb8c1a7a841d41a84fcdd8db7d0cdf3d78732c4f592" exitCode=60
Oct 30 22:00:03.533889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.533846    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" event={"ID":"d381869f-da79-449f-8a22-1970a5ef7dfa","Type":"ContainerDied","Data":"698d50a9b7f88adad586dbb8c1a7a841d41a84fcdd8db7d0cdf3d78732c4f592"}
Oct 30 22:00:03.534042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.534027    6695 scope.go:122] "RemoveContainer" containerID="698d50a9b7f88adad586dbb8c1a7a841d41a84fcdd8db7d0cdf3d78732c4f592"
Oct 30 22:00:03.536051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.536031    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerStarted","Data":"9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f"}
Oct 30 22:00:03.537767 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.537744    6695 generic.go:356] "Generic (PLEG): container finished" podID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerID="8d914d937af528a57bd17d31a6287b78a11f1921b7149fff893aff10b830171c" exitCode=60
Oct 30 22:00:03.537860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.537773    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" event={"ID":"3fa2035c-7a77-4bc7-9141-c912cf623256","Type":"ContainerDied","Data":"8d914d937af528a57bd17d31a6287b78a11f1921b7149fff893aff10b830171c"}
Oct 30 22:00:03.538058 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.538035    6695 scope.go:122] "RemoveContainer" containerID="8d914d937af528a57bd17d31a6287b78a11f1921b7149fff893aff10b830171c"
Oct 30 22:00:03.542941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.542915    6695 generic.go:356] "Generic (PLEG): container finished" podID="a9928370-f1a1-4126-9df2-376c65322ed3" containerID="d7d5ece9f3c82283e9e4fe5a32fe60f30de1972b2bda79c111dc718dd001aa6f" exitCode=60
Oct 30 22:00:03.543043 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.543022    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" event={"ID":"a9928370-f1a1-4126-9df2-376c65322ed3","Type":"ContainerDied","Data":"d7d5ece9f3c82283e9e4fe5a32fe60f30de1972b2bda79c111dc718dd001aa6f"}
Oct 30 22:00:03.543388 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.543364    6695 scope.go:122] "RemoveContainer" containerID="d7d5ece9f3c82283e9e4fe5a32fe60f30de1972b2bda79c111dc718dd001aa6f"
Oct 30 22:00:03.553863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.553074    6695 generic.go:356] "Generic (PLEG): container finished" podID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerID="a2bf948d69747946efd0c0dcabc8d59ce0c146b567f2310efa188c9438dec689" exitCode=60
Oct 30 22:00:03.553863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.553256    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" event={"ID":"f0df621f-cede-41ef-b8d0-b895f48034d5","Type":"ContainerDied","Data":"a2bf948d69747946efd0c0dcabc8d59ce0c146b567f2310efa188c9438dec689"}
Oct 30 22:00:03.553863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.553786    6695 scope.go:122] "RemoveContainer" containerID="a2bf948d69747946efd0c0dcabc8d59ce0c146b567f2310efa188c9438dec689"
Oct 30 22:00:03.558004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.557955    6695 generic.go:356] "Generic (PLEG): container finished" podID="c8257c09-556f-4890-a175-529173eaf1a8" containerID="cc73d7708b7055543c4218d4744d621acde7005c87a278b880e6d4cc22840bec" exitCode=60
Oct 30 22:00:03.558101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.558048    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerDied","Data":"cc73d7708b7055543c4218d4744d621acde7005c87a278b880e6d4cc22840bec"}
Oct 30 22:00:03.558879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.558854    6695 scope.go:122] "RemoveContainer" containerID="cc73d7708b7055543c4218d4744d621acde7005c87a278b880e6d4cc22840bec"
Oct 30 22:00:03.561998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.561955    6695 generic.go:356] "Generic (PLEG): container finished" podID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerID="974c1dd130352d7735a7f2c44e1e921377e07e4489d8d6e524479d6a03eb1ade" exitCode=60
Oct 30 22:00:03.562099 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.562031    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" event={"ID":"8193d2c6-b8f1-4460-adc6-5eefdda788d9","Type":"ContainerDied","Data":"974c1dd130352d7735a7f2c44e1e921377e07e4489d8d6e524479d6a03eb1ade"}
Oct 30 22:00:03.562530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.562510    6695 scope.go:122] "RemoveContainer" containerID="974c1dd130352d7735a7f2c44e1e921377e07e4489d8d6e524479d6a03eb1ade"
Oct 30 22:00:03.751162 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.751135    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:03.891191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:03.891132    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" podStartSLOduration=3.891113365 podStartE2EDuration="3.891113365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.844395036 +0000 UTC m=+2986.766853337" watchObservedRunningTime="2026-10-30 22:00:03.891113365 +0000 UTC m=+2986.813571666"
Oct 30 22:00:04.098797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.098764    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"]
Oct 30 22:00:04.215068 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:04.215033    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5619f6cb_592c_47c1_9bfa_d9f10ff261a0.slice/crio-804209ee9fb84fcb36882e5a5728ba26e0de99bcd73fabe7bd3d78c0ef46da7c WatchSource:0}: Error finding container 804209ee9fb84fcb36882e5a5728ba26e0de99bcd73fabe7bd3d78c0ef46da7c: Status 404 returned error can't find the container with id 804209ee9fb84fcb36882e5a5728ba26e0de99bcd73fabe7bd3d78c0ef46da7c
Oct 30 22:00:04.574002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.573965    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr" event={"ID":"5619f6cb-592c-47c1-9bfa-d9f10ff261a0","Type":"ContainerStarted","Data":"804209ee9fb84fcb36882e5a5728ba26e0de99bcd73fabe7bd3d78c0ef46da7c"}
Oct 30 22:00:04.576297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.576269    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerStarted","Data":"609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54"}
Oct 30 22:00:04.578287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.578259    6695 generic.go:356] "Generic (PLEG): container finished" podID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerID="9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f" exitCode=60
Oct 30 22:00:04.578381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.578341    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerDied","Data":"9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f"}
Oct 30 22:00:04.578452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.578396    6695 scope.go:122] "RemoveContainer" containerID="2cc1f6837ced31cf3b15bddca5a9d405b4b1c6e104fe2c3b5509ada77faf5af5"
Oct 30 22:00:04.578769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.578749    6695 scope.go:122] "RemoveContainer" containerID="9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f"
Oct 30 22:00:04.579171 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:04.579144    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx_odh-ai-gateway-infra(2b2b3430-038d-4ad5-9766-a9c51e1e7046)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046"
Oct 30 22:00:04.580521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.580496    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" event={"ID":"a9928370-f1a1-4126-9df2-376c65322ed3","Type":"ContainerStarted","Data":"a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b"}
Oct 30 22:00:04.582612 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.582565    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerStarted","Data":"f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996"}
Oct 30 22:00:04.584662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.584598    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" event={"ID":"8193d2c6-b8f1-4460-adc6-5eefdda788d9","Type":"ContainerStarted","Data":"99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f"}
Oct 30 22:00:04.586874 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.586850    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" event={"ID":"377405e1-8481-444f-a61a-5e59dcf38460","Type":"ContainerStarted","Data":"972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5"}
Oct 30 22:00:04.588934 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.588913    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerStarted","Data":"2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2"}
Oct 30 22:00:04.591035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.591007    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerStarted","Data":"f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a"}
Oct 30 22:00:04.628805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.628757    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" podStartSLOduration=4.628743115 podStartE2EDuration="4.628743115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.621695197 +0000 UTC m=+2987.544153495" watchObservedRunningTime="2026-10-30 22:00:04.628743115 +0000 UTC m=+2987.551201413"
Oct 30 22:00:04.699708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.699654    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" podStartSLOduration=4.699641012 podStartE2EDuration="4.699641012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.694587512 +0000 UTC m=+2987.617045809" watchObservedRunningTime="2026-10-30 22:00:04.699641012 +0000 UTC m=+2987.622099348"
Oct 30 22:00:04.739659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.739589    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" podStartSLOduration=4.739575418 podStartE2EDuration="4.739575418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.732880125 +0000 UTC m=+2987.655338420" watchObservedRunningTime="2026-10-30 22:00:04.739575418 +0000 UTC m=+2987.662033715"
Oct 30 22:00:04.774999 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.774939    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" podStartSLOduration=4.774925296 podStartE2EDuration="4.774925296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.77129314 +0000 UTC m=+2987.693751439" watchObservedRunningTime="2026-10-30 22:00:04.774925296 +0000 UTC m=+2987.697383594"
Oct 30 22:00:04.822494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.822427    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" podStartSLOduration=4.822405183 podStartE2EDuration="4.822405183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.809245595 +0000 UTC m=+2987.731703893" watchObservedRunningTime="2026-10-30 22:00:04.822405183 +0000 UTC m=+2987.744863484"
Oct 30 22:00:04.856140 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:04.856091    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" podStartSLOduration=4.856077415 podStartE2EDuration="4.856077415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.855147278 +0000 UTC m=+2987.777605576" watchObservedRunningTime="2026-10-30 22:00:04.856077415 +0000 UTC m=+2987.778535712"
Oct 30 22:00:05.602382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.602343    6695 generic.go:356] "Generic (PLEG): container finished" podID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerID="f7e503d1409189ac96271731dfb0eb1b353e288d76640402f06040fc248166d2" exitCode=60
Oct 30 22:00:05.602874 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.602430    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" event={"ID":"4ed668a4-c548-4e09-8d1d-09a08c500c37","Type":"ContainerDied","Data":"f7e503d1409189ac96271731dfb0eb1b353e288d76640402f06040fc248166d2"}
Oct 30 22:00:05.602874 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.602490    6695 scope.go:122] "RemoveContainer" containerID="13b393f644172d4a9e77464e74cf06281095e82beb7634280ba02a5ccc04ef0a"
Oct 30 22:00:05.603017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.602950    6695 scope.go:122] "RemoveContainer" containerID="f7e503d1409189ac96271731dfb0eb1b353e288d76640402f06040fc248166d2"
Oct 30 22:00:05.603409 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.603380    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk_odh-ai-gateway-infra(4ed668a4-c548-4e09-8d1d-09a08c500c37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37"
Oct 30 22:00:05.605429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.605389    6695 generic.go:356] "Generic (PLEG): container finished" podID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerID="75f8ab355c4844abaad496853fcc128033dd7894470f92c619249cd924a1ac4d" exitCode=60
Oct 30 22:00:05.605562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.605469    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" event={"ID":"c94a3b92-eb35-437d-8edb-5683bef4f768","Type":"ContainerDied","Data":"75f8ab355c4844abaad496853fcc128033dd7894470f92c619249cd924a1ac4d"}
Oct 30 22:00:05.605897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.605879    6695 scope.go:122] "RemoveContainer" containerID="75f8ab355c4844abaad496853fcc128033dd7894470f92c619249cd924a1ac4d"
Oct 30 22:00:05.606312 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.606289    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g_odh-ai-gateway-infra(c94a3b92-eb35-437d-8edb-5683bef4f768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768"
Oct 30 22:00:05.607631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.607595    6695 generic.go:356] "Generic (PLEG): container finished" podID="ed66b735-a7c1-470c-a775-11657620f643" containerID="20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b" exitCode=60
Oct 30 22:00:05.607755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.607710    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" event={"ID":"ed66b735-a7c1-470c-a775-11657620f643","Type":"ContainerDied","Data":"20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b"}
Oct 30 22:00:05.608062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.608044    6695 scope.go:122] "RemoveContainer" containerID="20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b"
Oct 30 22:00:05.608494 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.608470    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f_odh-ai-gateway-infra(ed66b735-a7c1-470c-a775-11657620f643)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" podUID="ed66b735-a7c1-470c-a775-11657620f643"
Oct 30 22:00:05.610879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.610856    6695 generic.go:356] "Generic (PLEG): container finished" podID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerID="74b05adfe06da1ec36968b9226b581ef27fee572602cfdbb538e18de5b16e3bb" exitCode=60
Oct 30 22:00:05.610976 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.610904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" event={"ID":"2a505c48-0f05-449a-bce4-570c9e6f82cd","Type":"ContainerDied","Data":"74b05adfe06da1ec36968b9226b581ef27fee572602cfdbb538e18de5b16e3bb"}
Oct 30 22:00:05.611234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.611215    6695 scope.go:122] "RemoveContainer" containerID="74b05adfe06da1ec36968b9226b581ef27fee572602cfdbb538e18de5b16e3bb"
Oct 30 22:00:05.611591 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.611569    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-tqlnx_odh-ai-gateway-infra(2a505c48-0f05-449a-bce4-570c9e6f82cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd"
Oct 30 22:00:05.612788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.612760    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr" event={"ID":"5619f6cb-592c-47c1-9bfa-d9f10ff261a0","Type":"ContainerStarted","Data":"a6310bb66a1c4ee194b615fdffa95c5b97e1f7071aaef93ba9c9f56570830138"}
Oct 30 22:00:05.614736 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.614717    6695 generic.go:356] "Generic (PLEG): container finished" podID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerID="609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54" exitCode=60
Oct 30 22:00:05.614832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.614798    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerDied","Data":"609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54"}
Oct 30 22:00:05.614998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.614984    6695 scope.go:122] "RemoveContainer" containerID="609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54"
Oct 30 22:00:05.615280 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.615258    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm_odh-ai-gateway-infra(ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f"
Oct 30 22:00:05.616875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.616845    6695 generic.go:356] "Generic (PLEG): container finished" podID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerID="4e20a4626d014d060e7fe18741f27eb6da40bee851e92c53824ad51506361167" exitCode=60
Oct 30 22:00:05.616981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.616880    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" event={"ID":"d381869f-da79-449f-8a22-1970a5ef7dfa","Type":"ContainerDied","Data":"4e20a4626d014d060e7fe18741f27eb6da40bee851e92c53824ad51506361167"}
Oct 30 22:00:05.617203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.617187    6695 scope.go:122] "RemoveContainer" containerID="4e20a4626d014d060e7fe18741f27eb6da40bee851e92c53824ad51506361167"
Oct 30 22:00:05.617509 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.617483    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b_odh-ai-gateway-infra(d381869f-da79-449f-8a22-1970a5ef7dfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa"
Oct 30 22:00:05.619066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.619048    6695 scope.go:122] "RemoveContainer" containerID="9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f"
Oct 30 22:00:05.619335 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.619313    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx_odh-ai-gateway-infra(2b2b3430-038d-4ad5-9766-a9c51e1e7046)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046"
Oct 30 22:00:05.620822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.620803    6695 generic.go:356] "Generic (PLEG): container finished" podID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerID="ab3fea4840be57c733a43683e44d55935ce9b00faadf1a1f1377e2ab63c67442" exitCode=60
Oct 30 22:00:05.620925 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.620880    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" event={"ID":"3fa2035c-7a77-4bc7-9141-c912cf623256","Type":"ContainerDied","Data":"ab3fea4840be57c733a43683e44d55935ce9b00faadf1a1f1377e2ab63c67442"}
Oct 30 22:00:05.621124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.621109    6695 scope.go:122] "RemoveContainer" containerID="ab3fea4840be57c733a43683e44d55935ce9b00faadf1a1f1377e2ab63c67442"
Oct 30 22:00:05.621385 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.621362    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r_odh-ai-gateway-infra(3fa2035c-7a77-4bc7-9141-c912cf623256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256"
Oct 30 22:00:05.622900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.622879    6695 generic.go:356] "Generic (PLEG): container finished" podID="a9928370-f1a1-4126-9df2-376c65322ed3" containerID="a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b" exitCode=60
Oct 30 22:00:05.622995 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.622962    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" event={"ID":"a9928370-f1a1-4126-9df2-376c65322ed3","Type":"ContainerDied","Data":"a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b"}
Oct 30 22:00:05.623177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.623160    6695 scope.go:122] "RemoveContainer" containerID="a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b"
Oct 30 22:00:05.623398 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.623376    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8_odh-ai-gateway-infra(a9928370-f1a1-4126-9df2-376c65322ed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" podUID="a9928370-f1a1-4126-9df2-376c65322ed3"
Oct 30 22:00:05.625063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.625041    6695 generic.go:356] "Generic (PLEG): container finished" podID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerID="3e05a0d8ff994be7f20b7d992d324cd5db84fdb7678f0ab801bcc2ba7cd10aa8" exitCode=60
Oct 30 22:00:05.625168 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.625122    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" event={"ID":"f0df621f-cede-41ef-b8d0-b895f48034d5","Type":"ContainerDied","Data":"3e05a0d8ff994be7f20b7d992d324cd5db84fdb7678f0ab801bcc2ba7cd10aa8"}
Oct 30 22:00:05.625323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.625306    6695 scope.go:122] "RemoveContainer" containerID="3e05a0d8ff994be7f20b7d992d324cd5db84fdb7678f0ab801bcc2ba7cd10aa8"
Oct 30 22:00:05.625551 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.625529    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22_odh-ai-gateway-infra(f0df621f-cede-41ef-b8d0-b895f48034d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5"
Oct 30 22:00:05.626949 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.626928    6695 generic.go:356] "Generic (PLEG): container finished" podID="c8257c09-556f-4890-a175-529173eaf1a8" containerID="f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996" exitCode=60
Oct 30 22:00:05.627054 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.627005    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerDied","Data":"f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996"}
Oct 30 22:00:05.627180 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.627166    6695 scope.go:122] "RemoveContainer" containerID="f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996"
Oct 30 22:00:05.627390 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.627370    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv_odh-ai-gateway-infra(c8257c09-556f-4890-a175-529173eaf1a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" podUID="c8257c09-556f-4890-a175-529173eaf1a8"
Oct 30 22:00:05.628697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.628677    6695 generic.go:356] "Generic (PLEG): container finished" podID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerID="99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f" exitCode=60
Oct 30 22:00:05.628769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.628720    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" event={"ID":"8193d2c6-b8f1-4460-adc6-5eefdda788d9","Type":"ContainerDied","Data":"99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f"}
Oct 30 22:00:05.628932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.628918    6695 scope.go:122] "RemoveContainer" containerID="99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f"
Oct 30 22:00:05.629181 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.629157    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j_odh-ai-gateway-infra(8193d2c6-b8f1-4460-adc6-5eefdda788d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9"
Oct 30 22:00:05.630706 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.630685    6695 generic.go:356] "Generic (PLEG): container finished" podID="377405e1-8481-444f-a61a-5e59dcf38460" containerID="972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5" exitCode=60
Oct 30 22:00:05.630801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.630785    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" event={"ID":"377405e1-8481-444f-a61a-5e59dcf38460","Type":"ContainerDied","Data":"972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5"}
Oct 30 22:00:05.630970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.630945    6695 scope.go:122] "RemoveContainer" containerID="972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5"
Oct 30 22:00:05.631171 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.631150    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd_odh-ai-gateway-infra(377405e1-8481-444f-a61a-5e59dcf38460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" podUID="377405e1-8481-444f-a61a-5e59dcf38460"
Oct 30 22:00:05.632693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.632667    6695 generic.go:356] "Generic (PLEG): container finished" podID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerID="2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2" exitCode=60
Oct 30 22:00:05.632765 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.632735    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerDied","Data":"2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2"}
Oct 30 22:00:05.632968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.632948    6695 scope.go:122] "RemoveContainer" containerID="2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2"
Oct 30 22:00:05.633208 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.633180    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk_odh-ai-gateway-infra(916d8393-b1f4-4a2c-a588-6d2f6e86cff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8"
Oct 30 22:00:05.634582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.634562    6695 generic.go:356] "Generic (PLEG): container finished" podID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerID="c48d17d2f7ad4f552df03403e110a7a79d22af72fc730fedfd7c6a3c3eacf1e3" exitCode=60
Oct 30 22:00:05.634685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.634649    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" event={"ID":"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc","Type":"ContainerDied","Data":"c48d17d2f7ad4f552df03403e110a7a79d22af72fc730fedfd7c6a3c3eacf1e3"}
Oct 30 22:00:05.634878 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.634864    6695 scope.go:122] "RemoveContainer" containerID="c48d17d2f7ad4f552df03403e110a7a79d22af72fc730fedfd7c6a3c3eacf1e3"
Oct 30 22:00:05.635102 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.635082    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v_odh-ai-gateway-infra(d7e1398e-f2fa-4b74-8525-a7e2017f2fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc"
Oct 30 22:00:05.636375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.636358    6695 generic.go:356] "Generic (PLEG): container finished" podID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerID="f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a" exitCode=60
Oct 30 22:00:05.636488 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.636387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerDied","Data":"f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a"}
Oct 30 22:00:05.636636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.636581    6695 scope.go:122] "RemoveContainer" containerID="f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a"
Oct 30 22:00:05.636832 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:05.636812    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc_odh-ai-gateway-infra(8df3ce61-1578-490a-b58b-3342bc5b1a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01"
Oct 30 22:00:05.881779 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.881752    6695 scope.go:122] "RemoveContainer" containerID="91fbd565980c8ac00346871cb737f384a54e7a4f55ff5ad78ebf0e26c585a9d3"
Oct 30 22:00:05.922257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.921504    6695 scope.go:122] "RemoveContainer" containerID="23e458c9044ce683925fd6055a9903270f96aae2b1c6345589541ca24e110dec"
Oct 30 22:00:05.939446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.939422    6695 scope.go:122] "RemoveContainer" containerID="9e4735254e6315a6c06706b16aa731c919f0b570d25761a34264d0235c83488e"
Oct 30 22:00:05.955461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.955431    6695 scope.go:122] "RemoveContainer" containerID="da1169f288bbf7ca47429d98c70b5664c9ccc055cd687353ca4e9784e30773bf"
Oct 30 22:00:05.971978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.971834    6695 scope.go:122] "RemoveContainer" containerID="698d50a9b7f88adad586dbb8c1a7a841d41a84fcdd8db7d0cdf3d78732c4f592"
Oct 30 22:00:05.986776 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:05.986755    6695 scope.go:122] "RemoveContainer" containerID="8d914d937af528a57bd17d31a6287b78a11f1921b7149fff893aff10b830171c"
Oct 30 22:00:06.001789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.001766    6695 scope.go:122] "RemoveContainer" containerID="d7d5ece9f3c82283e9e4fe5a32fe60f30de1972b2bda79c111dc718dd001aa6f"
Oct 30 22:00:06.018241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.018211    6695 scope.go:122] "RemoveContainer" containerID="a2bf948d69747946efd0c0dcabc8d59ce0c146b567f2310efa188c9438dec689"
Oct 30 22:00:06.033904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.033876    6695 scope.go:122] "RemoveContainer" containerID="cc73d7708b7055543c4218d4744d621acde7005c87a278b880e6d4cc22840bec"
Oct 30 22:00:06.108656 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.108596    6695 scope.go:122] "RemoveContainer" containerID="974c1dd130352d7735a7f2c44e1e921377e07e4489d8d6e524479d6a03eb1ade"
Oct 30 22:00:06.125424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.125393    6695 scope.go:122] "RemoveContainer" containerID="214a620ea26e616a74c2d21ffcbc2a1c4ec09e1ee1ad8a4390fe3854c92cd0c9"
Oct 30 22:00:06.141385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.141354    6695 scope.go:122] "RemoveContainer" containerID="2c01c6ee4652ed073cbaebcf32d1c7dee4156846c11be67b1023ea3a94644918"
Oct 30 22:00:06.157168 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.157146    6695 scope.go:122] "RemoveContainer" containerID="2ba3af77a2c5249efb1ff22d9318a3eeabf779ab97e0cce1276f90a8f27146f6"
Oct 30 22:00:06.172356 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.172333    6695 scope.go:122] "RemoveContainer" containerID="b6e07179c0012cf5b6d11cbf807e0af52ca10c1ead3d520ecb739eb7e7cc9563"
Oct 30 22:00:06.285834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.285780    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr" podStartSLOduration=3.285765083 podStartE2EDuration="3.285765083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.282215063 +0000 UTC m=+2989.204673361" watchObservedRunningTime="2026-10-30 22:00:06.285765083 +0000 UTC m=+2989.208223416"
Oct 30 22:00:06.656646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.656574    6695 scope.go:122] "RemoveContainer" containerID="20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b"
Oct 30 22:00:06.657177 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.657053    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f_odh-ai-gateway-infra(ed66b735-a7c1-470c-a775-11657620f643)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" podUID="ed66b735-a7c1-470c-a775-11657620f643"
Oct 30 22:00:06.658800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.658776    6695 scope.go:122] "RemoveContainer" containerID="74b05adfe06da1ec36968b9226b581ef27fee572602cfdbb538e18de5b16e3bb"
Oct 30 22:00:06.659170 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.659144    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-tqlnx_odh-ai-gateway-infra(2a505c48-0f05-449a-bce4-570c9e6f82cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd"
Oct 30 22:00:06.660477 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.660451    6695 generic.go:356] "Generic (PLEG): container finished" podID="5619f6cb-592c-47c1-9bfa-d9f10ff261a0" containerID="a6310bb66a1c4ee194b615fdffa95c5b97e1f7071aaef93ba9c9f56570830138" exitCode=0
Oct 30 22:00:06.660576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.660537    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr" event={"ID":"5619f6cb-592c-47c1-9bfa-d9f10ff261a0","Type":"ContainerDied","Data":"a6310bb66a1c4ee194b615fdffa95c5b97e1f7071aaef93ba9c9f56570830138"}
Oct 30 22:00:06.663009 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.662988    6695 scope.go:122] "RemoveContainer" containerID="609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54"
Oct 30 22:00:06.663370 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.663349    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm_odh-ai-gateway-infra(ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f"
Oct 30 22:00:06.665371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.665352    6695 scope.go:122] "RemoveContainer" containerID="4e20a4626d014d060e7fe18741f27eb6da40bee851e92c53824ad51506361167"
Oct 30 22:00:06.665741 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.665714    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b_odh-ai-gateway-infra(d381869f-da79-449f-8a22-1970a5ef7dfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa"
Oct 30 22:00:06.667062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.667044    6695 scope.go:122] "RemoveContainer" containerID="ab3fea4840be57c733a43683e44d55935ce9b00faadf1a1f1377e2ab63c67442"
Oct 30 22:00:06.667362 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.667342    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r_odh-ai-gateway-infra(3fa2035c-7a77-4bc7-9141-c912cf623256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256"
Oct 30 22:00:06.668692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.668674    6695 scope.go:122] "RemoveContainer" containerID="a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b"
Oct 30 22:00:06.668977 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.668952    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8_odh-ai-gateway-infra(a9928370-f1a1-4126-9df2-376c65322ed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" podUID="a9928370-f1a1-4126-9df2-376c65322ed3"
Oct 30 22:00:06.670283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.670268    6695 scope.go:122] "RemoveContainer" containerID="3e05a0d8ff994be7f20b7d992d324cd5db84fdb7678f0ab801bcc2ba7cd10aa8"
Oct 30 22:00:06.670486 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.670468    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22_odh-ai-gateway-infra(f0df621f-cede-41ef-b8d0-b895f48034d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5"
Oct 30 22:00:06.671864 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.671845    6695 scope.go:122] "RemoveContainer" containerID="f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996"
Oct 30 22:00:06.672086 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.672058    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv_odh-ai-gateway-infra(c8257c09-556f-4890-a175-529173eaf1a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" podUID="c8257c09-556f-4890-a175-529173eaf1a8"
Oct 30 22:00:06.679461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.679437    6695 scope.go:122] "RemoveContainer" containerID="99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f"
Oct 30 22:00:06.679791 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.679764    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j_odh-ai-gateway-infra(8193d2c6-b8f1-4460-adc6-5eefdda788d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9"
Oct 30 22:00:06.681913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.681886    6695 scope.go:122] "RemoveContainer" containerID="972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5"
Oct 30 22:00:06.682120 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.682102    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd_odh-ai-gateway-infra(377405e1-8481-444f-a61a-5e59dcf38460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" podUID="377405e1-8481-444f-a61a-5e59dcf38460"
Oct 30 22:00:06.683530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.683514    6695 scope.go:122] "RemoveContainer" containerID="2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2"
Oct 30 22:00:06.683743 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.683722    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk_odh-ai-gateway-infra(916d8393-b1f4-4a2c-a588-6d2f6e86cff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8"
Oct 30 22:00:06.685200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.685182    6695 scope.go:122] "RemoveContainer" containerID="c48d17d2f7ad4f552df03403e110a7a79d22af72fc730fedfd7c6a3c3eacf1e3"
Oct 30 22:00:06.685436 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.685411    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v_odh-ai-gateway-infra(d7e1398e-f2fa-4b74-8525-a7e2017f2fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc"
Oct 30 22:00:06.686945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.686926    6695 scope.go:122] "RemoveContainer" containerID="f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a"
Oct 30 22:00:06.687179 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.687147    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc_odh-ai-gateway-infra(8df3ce61-1578-490a-b58b-3342bc5b1a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01"
Oct 30 22:00:06.688675 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.688658    6695 scope.go:122] "RemoveContainer" containerID="f7e503d1409189ac96271731dfb0eb1b353e288d76640402f06040fc248166d2"
Oct 30 22:00:06.688897 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.688877    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk_odh-ai-gateway-infra(4ed668a4-c548-4e09-8d1d-09a08c500c37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37"
Oct 30 22:00:06.690190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:06.690167    6695 scope.go:122] "RemoveContainer" containerID="75f8ab355c4844abaad496853fcc128033dd7894470f92c619249cd924a1ac4d"
Oct 30 22:00:06.690359 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:06.690343    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g_odh-ai-gateway-infra(c94a3b92-eb35-437d-8edb-5683bef4f768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768"
Oct 30 22:00:07.484990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.484945    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6" podUID="dd615063-3ebf-4b00-ace6-0ee6f5a0076a" containerName="payload-processing" containerID="cri-o://3970c90ced84f73a05ee819ffc5b4252ac995a965d082ef5e659f18bd409ddaf" gracePeriod=25
Oct 30 22:00:07.704579 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.704535    6695 generic.go:356] "Generic (PLEG): container finished" podID="dd615063-3ebf-4b00-ace6-0ee6f5a0076a" containerID="3970c90ced84f73a05ee819ffc5b4252ac995a965d082ef5e659f18bd409ddaf" exitCode=0
Oct 30 22:00:07.704579 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.704579    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6" event={"ID":"dd615063-3ebf-4b00-ace6-0ee6f5a0076a","Type":"ContainerDied","Data":"3970c90ced84f73a05ee819ffc5b4252ac995a965d082ef5e659f18bd409ddaf"}
Oct 30 22:00:07.869749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.868623    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 22:00:07.982791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.982754    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials\") pod \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") "
Oct 30 22:00:07.983000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.982903    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume\") pod \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") "
Oct 30 22:00:07.983000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.982938    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume\") pod \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") "
Oct 30 22:00:07.983000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.982987    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll\") pod \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\" (UID: \"dd615063-3ebf-4b00-ace6-0ee6f5a0076a\") "
Oct 30 22:00:07.983513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.983369    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume" pod "dd615063-3ebf-4b00-ace6-0ee6f5a0076a" (UID: "dd615063-3ebf-4b00-ace6-0ee6f5a0076a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:07.983700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.983671    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:07.983794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.983709    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume" pod "dd615063-3ebf-4b00-ace6-0ee6f5a0076a" (UID: "dd615063-3ebf-4b00-ace6-0ee6f5a0076a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:07.985786 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.985751    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll" pod "dd615063-3ebf-4b00-ace6-0ee6f5a0076a" (UID: "dd615063-3ebf-4b00-ace6-0ee6f5a0076a"). InnerVolumeSpecName "kube-api-access-9d2ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:07.985899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.985794    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials" pod "dd615063-3ebf-4b00-ace6-0ee6f5a0076a" (UID: "dd615063-3ebf-4b00-ace6-0ee6f5a0076a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:07.996615 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:07.996579    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:08.084666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.084564    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca\") pod \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") "
Oct 30 22:00:08.084811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.084756    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r\" (UniqueName: \"kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r\") pod \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") "
Oct 30 22:00:08.084900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.084835    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca\") pod \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\" (UID: \"5619f6cb-592c-47c1-9bfa-d9f10ff261a0\") "
Oct 30 22:00:08.085221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.085193    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca" pod "5619f6cb-592c-47c1-9bfa-d9f10ff261a0" (UID: "5619f6cb-592c-47c1-9bfa-d9f10ff261a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:08.085550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.085529    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.085550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.085549    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.085735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.085559    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2ll\" (UniqueName: \"kubernetes.io/projected/dd615063-3ebf-4b00-ace6-0ee6f5a0076a-kube-api-access-9d2ll\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.085735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.085568    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.086711 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.086678    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca" pod "5619f6cb-592c-47c1-9bfa-d9f10ff261a0" (UID: "5619f6cb-592c-47c1-9bfa-d9f10ff261a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:08.087023 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.087001    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r" pod "5619f6cb-592c-47c1-9bfa-d9f10ff261a0" (UID: "5619f6cb-592c-47c1-9bfa-d9f10ff261a0"). InnerVolumeSpecName "kube-api-access-ttl6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:08.186962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.186925    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.186962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.186958    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttl6r\" (UniqueName: \"kubernetes.io/projected/5619f6cb-592c-47c1-9bfa-d9f10ff261a0-kube-api-access-ttl6r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:08.714689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.714656    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"
Oct 30 22:00:08.715106 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.714712    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr" event={"ID":"5619f6cb-592c-47c1-9bfa-d9f10ff261a0","Type":"ContainerDied","Data":"804209ee9fb84fcb36882e5a5728ba26e0de99bcd73fabe7bd3d78c0ef46da7c"}
Oct 30 22:00:08.715106 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.714768    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="804209ee9fb84fcb36882e5a5728ba26e0de99bcd73fabe7bd3d78c0ef46da7c"
Oct 30 22:00:08.716474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.716448    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"
Oct 30 22:00:08.716590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.716472    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6" event={"ID":"dd615063-3ebf-4b00-ace6-0ee6f5a0076a","Type":"ContainerDied","Data":"9cb70cb0b26f84537aa4a49aaef9e7efd50c7ccd28a2a0b1668c8036a009dd91"}
Oct 30 22:00:08.716590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.716536    6695 scope.go:122] "RemoveContainer" containerID="3970c90ced84f73a05ee819ffc5b4252ac995a965d082ef5e659f18bd409ddaf"
Oct 30 22:00:08.769835 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.769797    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"]
Oct 30 22:00:08.781213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:08.781143    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-55588v6t6"]
Oct 30 22:00:09.218047 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:09.217965    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd615063-3ebf-4b00-ace6-0ee6f5a0076a" path="/var/lib/kubelet/pods/dd615063-3ebf-4b00-ace6-0ee6f5a0076a/volumes"
Oct 30 22:00:14.354129 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:14.354090    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"]
Oct 30 22:00:16.207424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.207395    6695 scope.go:122] "RemoveContainer" containerID="f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996"
Oct 30 22:00:16.207919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.207595    6695 scope.go:122] "RemoveContainer" containerID="972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5"
Oct 30 22:00:16.207919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.207760    6695 scope.go:122] "RemoveContainer" containerID="609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54"
Oct 30 22:00:16.207919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.207897    6695 scope.go:122] "RemoveContainer" containerID="9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f"
Oct 30 22:00:16.208103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.208078    6695 scope.go:122] "RemoveContainer" containerID="2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2"
Oct 30 22:00:16.841066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.841037    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerStarted","Data":"546a2a17700d8f3923ec38f3c376eb2456815e3a0485da9896a04ed2f62d372c"}
Oct 30 22:00:16.843262 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.843230    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerStarted","Data":"44e2d8103aa23b2915dd35d96d8a1caaf987d0782d635f60f67ab8c86ac28ffb"}
Oct 30 22:00:16.845380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.845356    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerStarted","Data":"7bcdfec9d20a1a26ebb7d962c623d3b776ac9f97ef5c16bc27dd6e62f2ee4c41"}
Oct 30 22:00:16.847407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:16.847384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerStarted","Data":"807f2d8a900ccf6a259326878503e5b189ca0d8ce4efdb29027559cd92760a42"}
Oct 30 22:00:17.221228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.221143    6695 scope.go:122] "RemoveContainer" containerID="3e05a0d8ff994be7f20b7d992d324cd5db84fdb7678f0ab801bcc2ba7cd10aa8"
Oct 30 22:00:17.221648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.221346    6695 scope.go:122] "RemoveContainer" containerID="74b05adfe06da1ec36968b9226b581ef27fee572602cfdbb538e18de5b16e3bb"
Oct 30 22:00:17.221648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.221519    6695 scope.go:122] "RemoveContainer" containerID="4e20a4626d014d060e7fe18741f27eb6da40bee851e92c53824ad51506361167"
Oct 30 22:00:17.221754 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.221690    6695 scope.go:122] "RemoveContainer" containerID="a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b"
Oct 30 22:00:17.861384 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.861340    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" event={"ID":"f0df621f-cede-41ef-b8d0-b895f48034d5","Type":"ContainerStarted","Data":"865eb7e0ea68ffc5d62b8cd3b42fff3ef1612de74c546f23789971b05acaee2e"}
Oct 30 22:00:17.863780 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.863751    6695 generic.go:356] "Generic (PLEG): container finished" podID="c8257c09-556f-4890-a175-529173eaf1a8" containerID="546a2a17700d8f3923ec38f3c376eb2456815e3a0485da9896a04ed2f62d372c" exitCode=60
Oct 30 22:00:17.863932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.863826    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerDied","Data":"546a2a17700d8f3923ec38f3c376eb2456815e3a0485da9896a04ed2f62d372c"}
Oct 30 22:00:17.863932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.863883    6695 scope.go:122] "RemoveContainer" containerID="f6d9b48dfec53a266e39f6b89833186e061a3f7dbf342f9c112acd57d4a18996"
Oct 30 22:00:17.864299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.864279    6695 scope.go:122] "RemoveContainer" containerID="546a2a17700d8f3923ec38f3c376eb2456815e3a0485da9896a04ed2f62d372c"
Oct 30 22:00:17.864731 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:17.864704    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv_odh-ai-gateway-infra(c8257c09-556f-4890-a175-529173eaf1a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" podUID="c8257c09-556f-4890-a175-529173eaf1a8"
Oct 30 22:00:17.866015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.865993    6695 generic.go:356] "Generic (PLEG): container finished" podID="377405e1-8481-444f-a61a-5e59dcf38460" containerID="9a75c67f6ae046ee8d0f3415739cd65474d60070c2c0852dd82f8be05c56aaee" exitCode=60
Oct 30 22:00:17.866100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.866064    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" event={"ID":"377405e1-8481-444f-a61a-5e59dcf38460","Type":"ContainerDied","Data":"9a75c67f6ae046ee8d0f3415739cd65474d60070c2c0852dd82f8be05c56aaee"}
Oct 30 22:00:17.866479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.866460    6695 scope.go:122] "RemoveContainer" containerID="9a75c67f6ae046ee8d0f3415739cd65474d60070c2c0852dd82f8be05c56aaee"
Oct 30 22:00:17.866889 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:17.866865    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd_odh-ai-gateway-infra(377405e1-8481-444f-a61a-5e59dcf38460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" podUID="377405e1-8481-444f-a61a-5e59dcf38460"
Oct 30 22:00:17.868191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.868166    6695 generic.go:356] "Generic (PLEG): container finished" podID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerID="44e2d8103aa23b2915dd35d96d8a1caaf987d0782d635f60f67ab8c86ac28ffb" exitCode=60
Oct 30 22:00:17.868282 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.868203    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerDied","Data":"44e2d8103aa23b2915dd35d96d8a1caaf987d0782d635f60f67ab8c86ac28ffb"}
Oct 30 22:00:17.868617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.868588    6695 scope.go:122] "RemoveContainer" containerID="44e2d8103aa23b2915dd35d96d8a1caaf987d0782d635f60f67ab8c86ac28ffb"
Oct 30 22:00:17.868967 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:17.868941    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk_odh-ai-gateway-infra(916d8393-b1f4-4a2c-a588-6d2f6e86cff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8"
Oct 30 22:00:17.870578 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.870555    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" event={"ID":"2a505c48-0f05-449a-bce4-570c9e6f82cd","Type":"ContainerStarted","Data":"e78a1b66911454a6de3a7df8145d417999758fb8913d16483ccd1bbd0f521e2d"}
Oct 30 22:00:17.872916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.872894    6695 generic.go:356] "Generic (PLEG): container finished" podID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerID="7bcdfec9d20a1a26ebb7d962c623d3b776ac9f97ef5c16bc27dd6e62f2ee4c41" exitCode=60
Oct 30 22:00:17.873004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.872920    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerDied","Data":"7bcdfec9d20a1a26ebb7d962c623d3b776ac9f97ef5c16bc27dd6e62f2ee4c41"}
Oct 30 22:00:17.873241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.873226    6695 scope.go:122] "RemoveContainer" containerID="7bcdfec9d20a1a26ebb7d962c623d3b776ac9f97ef5c16bc27dd6e62f2ee4c41"
Oct 30 22:00:17.873465 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:17.873444    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm_odh-ai-gateway-infra(ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f"
Oct 30 22:00:17.875125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.875104    6695 generic.go:356] "Generic (PLEG): container finished" podID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerID="807f2d8a900ccf6a259326878503e5b189ca0d8ce4efdb29027559cd92760a42" exitCode=60
Oct 30 22:00:17.875240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.875158    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerDied","Data":"807f2d8a900ccf6a259326878503e5b189ca0d8ce4efdb29027559cd92760a42"}
Oct 30 22:00:17.875429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.875412    6695 scope.go:122] "RemoveContainer" containerID="807f2d8a900ccf6a259326878503e5b189ca0d8ce4efdb29027559cd92760a42"
Oct 30 22:00:17.875660 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:17.875643    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx_odh-ai-gateway-infra(2b2b3430-038d-4ad5-9766-a9c51e1e7046)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046"
Oct 30 22:00:17.880475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.880450    6695 scope.go:122] "RemoveContainer" containerID="972fe0e2c526f8ab656315640054e082f3783803c8bba21078dfff9635e147f5"
Oct 30 22:00:17.891410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.891383    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"]
Oct 30 22:00:17.897359 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.897340    6695 scope.go:122] "RemoveContainer" containerID="2742d8df361e54e6ec1bae8c7c5a083f0076824a310062e42f0e8bb23d5980e2"
Oct 30 22:00:17.903163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.903115    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" podStartSLOduration=17.903101848 podStartE2EDuration="17.903101848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:17.899055265 +0000 UTC m=+3000.821513564" watchObservedRunningTime="2026-10-30 22:00:17.903101848 +0000 UTC m=+3000.825560137"
Oct 30 22:00:17.918187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.918162    6695 scope.go:122] "RemoveContainer" containerID="609ea0a6cea9ed0ea617bf29d877091b1ffb766a8657eaca3d23403b90d48c54"
Oct 30 22:00:17.933530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.933498    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"]
Oct 30 22:00:17.935648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.935627    6695 scope.go:122] "RemoveContainer" containerID="9e5716a8eadfeb37fadfa06aee74bf088ad25336a843dab5109e68cb6cbf1f9f"
Oct 30 22:00:17.972623 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.972580    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"]
Oct 30 22:00:17.980202 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:17.980097    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" podStartSLOduration=17.980079318 podStartE2EDuration="17.980079318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:17.97767154 +0000 UTC m=+3000.900129841" watchObservedRunningTime="2026-10-30 22:00:17.980079318 +0000 UTC m=+3000.902537626"
Oct 30 22:00:18.013450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.013411    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"]
Oct 30 22:00:18.206873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.206777    6695 scope.go:122] "RemoveContainer" containerID="f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a"
Oct 30 22:00:18.207030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.206931    6695 scope.go:122] "RemoveContainer" containerID="ab3fea4840be57c733a43683e44d55935ce9b00faadf1a1f1377e2ab63c67442"
Oct 30 22:00:18.207132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.207110    6695 scope.go:122] "RemoveContainer" containerID="f7e503d1409189ac96271731dfb0eb1b353e288d76640402f06040fc248166d2"
Oct 30 22:00:18.207267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.207246    6695 scope.go:122] "RemoveContainer" containerID="75f8ab355c4844abaad496853fcc128033dd7894470f92c619249cd924a1ac4d"
Oct 30 22:00:18.207381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.207356    6695 scope.go:122] "RemoveContainer" containerID="20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b"
Oct 30 22:00:18.890434 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.890403    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" event={"ID":"4ed668a4-c548-4e09-8d1d-09a08c500c37","Type":"ContainerStarted","Data":"876bf6365d6418ed87fe2e11d8b7e01580b950dcf0d3222b96dbcd70cfd53eb5"}
Oct 30 22:00:18.892678 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.892651    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" event={"ID":"c94a3b92-eb35-437d-8edb-5683bef4f768","Type":"ContainerStarted","Data":"9f52d357c3881ad1a9d240e3ba6364f642d9dabcdcca1d81f68840dbf6e9a738"}
Oct 30 22:00:18.894363 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.894339    6695 generic.go:356] "Generic (PLEG): container finished" podID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerID="e78a1b66911454a6de3a7df8145d417999758fb8913d16483ccd1bbd0f521e2d" exitCode=60
Oct 30 22:00:18.894469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.894413    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" event={"ID":"2a505c48-0f05-449a-bce4-570c9e6f82cd","Type":"ContainerDied","Data":"e78a1b66911454a6de3a7df8145d417999758fb8913d16483ccd1bbd0f521e2d"}
Oct 30 22:00:18.894536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.894475    6695 scope.go:122] "RemoveContainer" containerID="74b05adfe06da1ec36968b9226b581ef27fee572602cfdbb538e18de5b16e3bb"
Oct 30 22:00:18.894817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.894799    6695 scope.go:122] "RemoveContainer" containerID="e78a1b66911454a6de3a7df8145d417999758fb8913d16483ccd1bbd0f521e2d"
Oct 30 22:00:18.895108 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:18.895087    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-tqlnx_odh-ai-gateway-infra(2a505c48-0f05-449a-bce4-570c9e6f82cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd"
Oct 30 22:00:18.899099 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.899074    6695 generic.go:356] "Generic (PLEG): container finished" podID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerID="7dc53d3a1bc4c10850b99640e9fb1bc208b0a421700a359e0fbb9fd3ee577067" exitCode=60
Oct 30 22:00:18.899198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.899156    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" event={"ID":"d381869f-da79-449f-8a22-1970a5ef7dfa","Type":"ContainerDied","Data":"7dc53d3a1bc4c10850b99640e9fb1bc208b0a421700a359e0fbb9fd3ee577067"}
Oct 30 22:00:18.899756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.899737    6695 scope.go:122] "RemoveContainer" containerID="7dc53d3a1bc4c10850b99640e9fb1bc208b0a421700a359e0fbb9fd3ee577067"
Oct 30 22:00:18.900077 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:18.900054    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b_odh-ai-gateway-infra(d381869f-da79-449f-8a22-1970a5ef7dfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa"
Oct 30 22:00:18.904155 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.904122    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" event={"ID":"3fa2035c-7a77-4bc7-9141-c912cf623256","Type":"ContainerStarted","Data":"de727a0ff06c87b9ab217a7fbf8af76f34122bab50ca408762294378eaab166b"}
Oct 30 22:00:18.906498 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.906474    6695 generic.go:356] "Generic (PLEG): container finished" podID="a9928370-f1a1-4126-9df2-376c65322ed3" containerID="691d95016be5c5ba04cc05a136df8e7b9de7d4af44abfd3b20738e992b2643dd" exitCode=60
Oct 30 22:00:18.906641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.906502    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" event={"ID":"a9928370-f1a1-4126-9df2-376c65322ed3","Type":"ContainerDied","Data":"691d95016be5c5ba04cc05a136df8e7b9de7d4af44abfd3b20738e992b2643dd"}
Oct 30 22:00:18.907051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.906931    6695 scope.go:122] "RemoveContainer" containerID="691d95016be5c5ba04cc05a136df8e7b9de7d4af44abfd3b20738e992b2643dd"
Oct 30 22:00:18.907289 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:18.907245    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8_odh-ai-gateway-infra(a9928370-f1a1-4126-9df2-376c65322ed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" podUID="a9928370-f1a1-4126-9df2-376c65322ed3"
Oct 30 22:00:18.910973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.910855    6695 generic.go:356] "Generic (PLEG): container finished" podID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerID="865eb7e0ea68ffc5d62b8cd3b42fff3ef1612de74c546f23789971b05acaee2e" exitCode=60
Oct 30 22:00:18.910973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.910880    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" event={"ID":"f0df621f-cede-41ef-b8d0-b895f48034d5","Type":"ContainerDied","Data":"865eb7e0ea68ffc5d62b8cd3b42fff3ef1612de74c546f23789971b05acaee2e"}
Oct 30 22:00:18.911352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.911273    6695 scope.go:122] "RemoveContainer" containerID="865eb7e0ea68ffc5d62b8cd3b42fff3ef1612de74c546f23789971b05acaee2e"
Oct 30 22:00:18.911784 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:18.911666    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22_odh-ai-gateway-infra(f0df621f-cede-41ef-b8d0-b895f48034d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5"
Oct 30 22:00:18.921327 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.921216    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"]
Oct 30 22:00:18.943885 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.943552    6695 scope.go:122] "RemoveContainer" containerID="4e20a4626d014d060e7fe18741f27eb6da40bee851e92c53824ad51506361167"
Oct 30 22:00:18.973946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:18.973809    6695 scope.go:122] "RemoveContainer" containerID="a4ebd7b5dd12aba72d114deb493cdb1daa2ab0f7d4177029aaaf21722a25df4b"
Oct 30 22:00:19.015635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.011504    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"]
Oct 30 22:00:19.051177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.049176    6695 scope.go:122] "RemoveContainer" containerID="3e05a0d8ff994be7f20b7d992d324cd5db84fdb7678f0ab801bcc2ba7cd10aa8"
Oct 30 22:00:19.146098 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.146073    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:19.154103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.154067    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"]
Oct 30 22:00:19.173649 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.167047    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" podStartSLOduration=19.167009851 podStartE2EDuration="19.167009851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:19.165677558 +0000 UTC m=+3002.088135857" watchObservedRunningTime="2026-10-30 22:00:19.167009851 +0000 UTC m=+3002.089468150"
Oct 30 22:00:19.197482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.197449    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca\") pod \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") "
Oct 30 22:00:19.197667 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.197636    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg\" (UniqueName: \"kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg\") pod \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\" (UID: \"916d8393-b1f4-4a2c-a588-6d2f6e86cff8\") "
Oct 30 22:00:19.202502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.198796    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca" pod "916d8393-b1f4-4a2c-a588-6d2f6e86cff8" (UID: "916d8393-b1f4-4a2c-a588-6d2f6e86cff8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.202502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.202389    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" podStartSLOduration=19.202376959 podStartE2EDuration="19.202376959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:19.201796223 +0000 UTC m=+3002.124254521" watchObservedRunningTime="2026-10-30 22:00:19.202376959 +0000 UTC m=+3002.124835253"
Oct 30 22:00:19.205962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.205932    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg" pod "916d8393-b1f4-4a2c-a588-6d2f6e86cff8" (UID: "916d8393-b1f4-4a2c-a588-6d2f6e86cff8"). InnerVolumeSpecName "kube-api-access-sptxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.259503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.259464    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:19.267008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.266104    6695 scope.go:122] "RemoveContainer" containerID="99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f"
Oct 30 22:00:19.267008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.266572    6695 scope.go:122] "RemoveContainer" containerID="c48d17d2f7ad4f552df03403e110a7a79d22af72fc730fedfd7c6a3c3eacf1e3"
Oct 30 22:00:19.299679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.298628    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls\" (UniqueName: \"kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls\") pod \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") "
Oct 30 22:00:19.299679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.298913    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca\") pod \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\" (UID: \"2b2b3430-038d-4ad5-9766-a9c51e1e7046\") "
Oct 30 22:00:19.301277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.301256    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.301505 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.301488    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sptxg\" (UniqueName: \"kubernetes.io/projected/916d8393-b1f4-4a2c-a588-6d2f6e86cff8-kube-api-access-sptxg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.302422 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.302397    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca" pod "2b2b3430-038d-4ad5-9766-a9c51e1e7046" (UID: "2b2b3430-038d-4ad5-9766-a9c51e1e7046"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.304133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.304105    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls" pod "2b2b3430-038d-4ad5-9766-a9c51e1e7046" (UID: "2b2b3430-038d-4ad5-9766-a9c51e1e7046"). InnerVolumeSpecName "kube-api-access-pq2ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.345759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.345588    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" podStartSLOduration=19.345570097 podStartE2EDuration="19.345570097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:19.344118328 +0000 UTC m=+3002.266576638" watchObservedRunningTime="2026-10-30 22:00:19.345570097 +0000 UTC m=+3002.268028396"
Oct 30 22:00:19.354777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.354732    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc" podUID="51e81593-4356-4296-9736-9dae8c77fb77" containerName="payload-processing" containerID="cri-o://11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9" gracePeriod=25
Oct 30 22:00:19.403528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.403495    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2b3430-038d-4ad5-9766-a9c51e1e7046-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.403528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.403526    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2ls\" (UniqueName: \"kubernetes.io/projected/2b2b3430-038d-4ad5-9766-a9c51e1e7046-kube-api-access-pq2ls\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.832590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.832558    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:19.869803 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.869774    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:19.879503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.879479    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:19.881864 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.881841    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 22:00:19.936322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936274    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials\") pod \"51e81593-4356-4296-9736-9dae8c77fb77\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936377    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz\" (UniqueName: \"kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz\") pod \"377405e1-8481-444f-a61a-5e59dcf38460\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936482    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd\" (UniqueName: \"kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd\") pod \"c8257c09-556f-4890-a175-529173eaf1a8\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936519    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca\") pod \"377405e1-8481-444f-a61a-5e59dcf38460\" (UID: \"377405e1-8481-444f-a61a-5e59dcf38460\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936551    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume\") pod \"51e81593-4356-4296-9736-9dae8c77fb77\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936732    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5\" (UniqueName: \"kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5\") pod \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936765    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca\") pod \"c8257c09-556f-4890-a175-529173eaf1a8\" (UID: \"c8257c09-556f-4890-a175-529173eaf1a8\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936807    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz\") pod \"51e81593-4356-4296-9736-9dae8c77fb77\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") "
Oct 30 22:00:19.936912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936843    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume\") pod \"51e81593-4356-4296-9736-9dae8c77fb77\" (UID: \"51e81593-4356-4296-9736-9dae8c77fb77\") "
Oct 30 22:00:19.937494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936938    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca\") pod \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\" (UID: \"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f\") "
Oct 30 22:00:19.937494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.936991    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume" pod "51e81593-4356-4296-9736-9dae8c77fb77" (UID: "51e81593-4356-4296-9736-9dae8c77fb77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.937726 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.937704    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca" pod "c8257c09-556f-4890-a175-529173eaf1a8" (UID: "c8257c09-556f-4890-a175-529173eaf1a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.938081 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.937888    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca" pod "ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" (UID: "ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.938081 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.937983    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume" pod "51e81593-4356-4296-9736-9dae8c77fb77" (UID: "51e81593-4356-4296-9736-9dae8c77fb77"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.938288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.938179    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8257c09-556f-4890-a175-529173eaf1a8-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.938288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.938228    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.938288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.938245    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.938288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.938261    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e81593-4356-4296-9736-9dae8c77fb77-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:19.938587 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.938552    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca" pod "377405e1-8481-444f-a61a-5e59dcf38460" (UID: "377405e1-8481-444f-a61a-5e59dcf38460"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.941462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.941224    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials" pod "51e81593-4356-4296-9736-9dae8c77fb77" (UID: "51e81593-4356-4296-9736-9dae8c77fb77"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.941462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.941306    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd" pod "c8257c09-556f-4890-a175-529173eaf1a8" (UID: "c8257c09-556f-4890-a175-529173eaf1a8"). InnerVolumeSpecName "kube-api-access-5d9bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.941697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.941501    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz" pod "51e81593-4356-4296-9736-9dae8c77fb77" (UID: "51e81593-4356-4296-9736-9dae8c77fb77"). InnerVolumeSpecName "kube-api-access-dcfvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.941883 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.941853    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz" pod "377405e1-8481-444f-a61a-5e59dcf38460" (UID: "377405e1-8481-444f-a61a-5e59dcf38460"). InnerVolumeSpecName "kube-api-access-ztzxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.942355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.942335    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5" pod "ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" (UID: "ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f"). InnerVolumeSpecName "kube-api-access-p4bk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.947354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.947330    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv" event={"ID":"c8257c09-556f-4890-a175-529173eaf1a8","Type":"ContainerDied","Data":"fd2b7a4edf8bbd3cad9b4ca06bce009714731072251b55f951f5d02377c65d09"}
Oct 30 22:00:19.947455 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.947410    6695 scope.go:122] "RemoveContainer" containerID="546a2a17700d8f3923ec38f3c376eb2456815e3a0485da9896a04ed2f62d372c"
Oct 30 22:00:19.947527 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.947513    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"
Oct 30 22:00:19.970971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.970939    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"
Oct 30 22:00:19.971185 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.970932    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd" event={"ID":"377405e1-8481-444f-a61a-5e59dcf38460","Type":"ContainerDied","Data":"d86fbec8fc07c37036c9402650d86034dc62ba2c2aebf8af97b578950fc6f073"}
Oct 30 22:00:19.977791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.977058    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"
Oct 30 22:00:19.981637 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.979441    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk" event={"ID":"916d8393-b1f4-4a2c-a588-6d2f6e86cff8","Type":"ContainerDied","Data":"8dc16c909c47e6a679f61e1e113daf43f52de69084bd11697feea749466a2426"}
Oct 30 22:00:19.995051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.995027    6695 scope.go:122] "RemoveContainer" containerID="9a75c67f6ae046ee8d0f3415739cd65474d60070c2c0852dd82f8be05c56aaee"
Oct 30 22:00:19.998494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.997681    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"]
Oct 30 22:00:19.999968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:19.999943    6695 generic.go:356] "Generic (PLEG): container finished" podID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerID="c09afb93c86ecabaff9173fc930b5f61c66b65fe28e810d588cec8764125cbb5" exitCode=60
Oct 30 22:00:20.000089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.000073    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerDied","Data":"c09afb93c86ecabaff9173fc930b5f61c66b65fe28e810d588cec8764125cbb5"}
Oct 30 22:00:20.000567 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.000542    6695 scope.go:122] "RemoveContainer" containerID="c09afb93c86ecabaff9173fc930b5f61c66b65fe28e810d588cec8764125cbb5"
Oct 30 22:00:20.001029 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.001001    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc_odh-ai-gateway-infra(8df3ce61-1578-490a-b58b-3342bc5b1a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01"
Oct 30 22:00:20.006126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.006086    6695 generic.go:356] "Generic (PLEG): container finished" podID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerID="876bf6365d6418ed87fe2e11d8b7e01580b950dcf0d3222b96dbcd70cfd53eb5" exitCode=60
Oct 30 22:00:20.006400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.006379    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" event={"ID":"4ed668a4-c548-4e09-8d1d-09a08c500c37","Type":"ContainerDied","Data":"876bf6365d6418ed87fe2e11d8b7e01580b950dcf0d3222b96dbcd70cfd53eb5"}
Oct 30 22:00:20.008194 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.008163    6695 scope.go:122] "RemoveContainer" containerID="876bf6365d6418ed87fe2e11d8b7e01580b950dcf0d3222b96dbcd70cfd53eb5"
Oct 30 22:00:20.017443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.017362    6695 scope.go:122] "RemoveContainer" containerID="44e2d8103aa23b2915dd35d96d8a1caaf987d0782d635f60f67ab8c86ac28ffb"
Oct 30 22:00:20.018093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.018070    6695 generic.go:356] "Generic (PLEG): container finished" podID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerID="9f52d357c3881ad1a9d240e3ba6364f642d9dabcdcca1d81f68840dbf6e9a738" exitCode=60
Oct 30 22:00:20.018199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.018153    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" event={"ID":"c94a3b92-eb35-437d-8edb-5683bef4f768","Type":"ContainerDied","Data":"9f52d357c3881ad1a9d240e3ba6364f642d9dabcdcca1d81f68840dbf6e9a738"}
Oct 30 22:00:20.019728 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.019700    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk_odh-ai-gateway-infra(4ed668a4-c548-4e09-8d1d-09a08c500c37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37"
Oct 30 22:00:20.020961 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.020292    6695 scope.go:122] "RemoveContainer" containerID="9f52d357c3881ad1a9d240e3ba6364f642d9dabcdcca1d81f68840dbf6e9a738"
Oct 30 22:00:20.022333 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.022287    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g_odh-ai-gateway-infra(c94a3b92-eb35-437d-8edb-5683bef4f768)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768"
Oct 30 22:00:20.029663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.029421    6695 generic.go:356] "Generic (PLEG): container finished" podID="ed66b735-a7c1-470c-a775-11657620f643" containerID="2f4712f43ce17c93542289e7bab24eb516f591f3fd349887b56d3c14ab7c64c4" exitCode=60
Oct 30 22:00:20.029663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.029541    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" event={"ID":"ed66b735-a7c1-470c-a775-11657620f643","Type":"ContainerDied","Data":"2f4712f43ce17c93542289e7bab24eb516f591f3fd349887b56d3c14ab7c64c4"}
Oct 30 22:00:20.032451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.030950    6695 scope.go:122] "RemoveContainer" containerID="2f4712f43ce17c93542289e7bab24eb516f591f3fd349887b56d3c14ab7c64c4"
Oct 30 22:00:20.032451 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.031333    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f_odh-ai-gateway-infra(ed66b735-a7c1-470c-a775-11657620f643)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" podUID="ed66b735-a7c1-470c-a775-11657620f643"
Oct 30 22:00:20.036132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.036104    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm" event={"ID":"ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f","Type":"ContainerDied","Data":"5443fb9c7cdeb89f6aa20c281d8e59aeda2a4ddce3ddb35122a60638667fb3c0"}
Oct 30 22:00:20.036231 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.036205    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"
Oct 30 22:00:20.038163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.038043    6695 scope.go:122] "RemoveContainer" containerID="f1f5199282c6d552bc71efdbba7fc3b051f60616aa7327f60273d7e65b879e0a"
Oct 30 22:00:20.047136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.044504    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bk5\" (UniqueName: \"kubernetes.io/projected/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f-kube-api-access-p4bk5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.047136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.044528    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfvz\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-kube-api-access-dcfvz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.047136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.044544    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51e81593-4356-4296-9736-9dae8c77fb77-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.047136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.044561    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzxz\" (UniqueName: \"kubernetes.io/projected/377405e1-8481-444f-a61a-5e59dcf38460-kube-api-access-ztzxz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.047136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.044577    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9bd\" (UniqueName: \"kubernetes.io/projected/c8257c09-556f-4890-a175-529173eaf1a8-kube-api-access-5d9bd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.047136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.044592    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377405e1-8481-444f-a61a-5e59dcf38460-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.052655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.052289    6695 generic.go:356] "Generic (PLEG): container finished" podID="51e81593-4356-4296-9736-9dae8c77fb77" containerID="11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9" exitCode=0
Oct 30 22:00:20.052655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.052381    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc" event={"ID":"51e81593-4356-4296-9736-9dae8c77fb77","Type":"ContainerDied","Data":"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9"}
Oct 30 22:00:20.052655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.052414    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc" event={"ID":"51e81593-4356-4296-9736-9dae8c77fb77","Type":"ContainerDied","Data":"10b959f6c2e34f6a3302b5c9015951214b96312cb4c45c8fa53e3d63db63c0c4"}
Oct 30 22:00:20.052655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.052464    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"
Oct 30 22:00:20.058150 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.058124    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx" event={"ID":"2b2b3430-038d-4ad5-9766-a9c51e1e7046","Type":"ContainerDied","Data":"7de5c13dfa1510aad7c8b9f3f40fe04ead96fc072c0cc2d24497329312802b35"}
Oct 30 22:00:20.058253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.058234    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"
Oct 30 22:00:20.064344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.063001    6695 generic.go:356] "Generic (PLEG): container finished" podID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerID="de727a0ff06c87b9ab217a7fbf8af76f34122bab50ca408762294378eaab166b" exitCode=60
Oct 30 22:00:20.064344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.063053    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" event={"ID":"3fa2035c-7a77-4bc7-9141-c912cf623256","Type":"ContainerDied","Data":"de727a0ff06c87b9ab217a7fbf8af76f34122bab50ca408762294378eaab166b"}
Oct 30 22:00:20.064344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.064087    6695 scope.go:122] "RemoveContainer" containerID="de727a0ff06c87b9ab217a7fbf8af76f34122bab50ca408762294378eaab166b"
Oct 30 22:00:20.064938 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.064914    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r_odh-ai-gateway-infra(3fa2035c-7a77-4bc7-9141-c912cf623256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256"
Oct 30 22:00:20.068311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.068288    6695 scope.go:122] "RemoveContainer" containerID="f7e503d1409189ac96271731dfb0eb1b353e288d76640402f06040fc248166d2"
Oct 30 22:00:20.090120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.090095    6695 scope.go:122] "RemoveContainer" containerID="75f8ab355c4844abaad496853fcc128033dd7894470f92c619249cd924a1ac4d"
Oct 30 22:00:20.108007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.107939    6695 scope.go:122] "RemoveContainer" containerID="20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b"
Oct 30 22:00:20.133429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.133404    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"]
Oct 30 22:00:20.150628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.150585    6695 scope.go:122] "RemoveContainer" containerID="7bcdfec9d20a1a26ebb7d962c623d3b776ac9f97ef5c16bc27dd6e62f2ee4c41"
Oct 30 22:00:20.155537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.155504    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3172e5-29889960-7wcpk"]
Oct 30 22:00:20.172406 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.172369    6695 scope.go:122] "RemoveContainer" containerID="11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9"
Oct 30 22:00:20.174083 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.174054    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b\": container with ID starting with 20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b not found: ID does not exist" containerID="20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b"
Oct 30 22:00:20.174266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.174241    6695 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b\": container with ID starting with 20119e8d311c612a43b67e9aa4e7619e56e041840ac989d61add50a95dbc872b not found: ID does not exist"
Oct 30 22:00:20.184757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.184724    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"]
Oct 30 22:00:20.205184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.198768    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"]
Oct 30 22:00:20.217217 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.210719    6695 scope.go:122] "RemoveContainer" containerID="11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9"
Oct 30 22:00:20.217315 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.217280    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9\": container with ID starting with 11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9 not found: ID does not exist" containerID="11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9"
Oct 30 22:00:20.217371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.217319    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9"} err="failed to get container status \"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9\": rpc error: code = NotFound desc = could not find container \"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9\": container with ID starting with 11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9 not found: ID does not exist"
Oct 30 22:00:20.217371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.217344    6695 scope.go:122] "RemoveContainer" containerID="807f2d8a900ccf6a259326878503e5b189ca0d8ce4efdb29027559cd92760a42"
Oct 30 22:00:20.218152 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:20.218027    6695 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9\": container with ID starting with 11756e686cf7be5ba9a8a049e75cb4b336cd6bfeb0cac93b984064df4360daf9 not found: ID does not exist"
Oct 30 22:00:20.227797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.227711    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdb168-29889960-w6lbd"]
Oct 30 22:00:20.233777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.233711    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"]
Oct 30 22:00:20.244322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.244204    6695 scope.go:122] "RemoveContainer" containerID="ab3fea4840be57c733a43683e44d55935ce9b00faadf1a1f1377e2ab63c67442"
Oct 30 22:00:20.273818 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.273744    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"]
Oct 30 22:00:20.283310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.283279    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-db1288-29889960-grtmv"]
Oct 30 22:00:20.361700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.361664    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"]
Oct 30 22:00:20.395417 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.395388    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"]
Oct 30 22:00:20.395925 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.395826    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:20.400398 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.400340    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:20.404116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.404089    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:20.420027 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.419995    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"]
Oct 30 22:00:20.424739 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.424694    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-7b7f64dxc"]
Oct 30 22:00:20.458171 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458138    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"]
Oct 30 22:00:20.458399 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458317    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca\") pod \"d381869f-da79-449f-8a22-1970a5ef7dfa\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") "
Oct 30 22:00:20.458399 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458390    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r\" (UniqueName: \"kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r\") pod \"f0df621f-cede-41ef-b8d0-b895f48034d5\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") "
Oct 30 22:00:20.458536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458435    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca\") pod \"f0df621f-cede-41ef-b8d0-b895f48034d5\" (UID: \"f0df621f-cede-41ef-b8d0-b895f48034d5\") "
Oct 30 22:00:20.458536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458482    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24\" (UniqueName: \"kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24\") pod \"2a505c48-0f05-449a-bce4-570c9e6f82cd\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") "
Oct 30 22:00:20.458675 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458587    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca\") pod \"2a505c48-0f05-449a-bce4-570c9e6f82cd\" (UID: \"2a505c48-0f05-449a-bce4-570c9e6f82cd\") "
Oct 30 22:00:20.458774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458760    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n\" (UniqueName: \"kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n\") pod \"d381869f-da79-449f-8a22-1970a5ef7dfa\" (UID: \"d381869f-da79-449f-8a22-1970a5ef7dfa\") "
Oct 30 22:00:20.458956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.458930    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca" pod "f0df621f-cede-41ef-b8d0-b895f48034d5" (UID: "f0df621f-cede-41ef-b8d0-b895f48034d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.459414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.459384    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0df621f-cede-41ef-b8d0-b895f48034d5-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.459913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.459810    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca" pod "d381869f-da79-449f-8a22-1970a5ef7dfa" (UID: "d381869f-da79-449f-8a22-1970a5ef7dfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.460078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.460053    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca" pod "2a505c48-0f05-449a-bce4-570c9e6f82cd" (UID: "2a505c48-0f05-449a-bce4-570c9e6f82cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.461811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.461782    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n" pod "d381869f-da79-449f-8a22-1970a5ef7dfa" (UID: "d381869f-da79-449f-8a22-1970a5ef7dfa"). InnerVolumeSpecName "kube-api-access-d6g6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.461915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.461822    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r" pod "f0df621f-cede-41ef-b8d0-b895f48034d5" (UID: "f0df621f-cede-41ef-b8d0-b895f48034d5"). InnerVolumeSpecName "kube-api-access-khd2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.461915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.461906    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24" pod "2a505c48-0f05-449a-bce4-570c9e6f82cd" (UID: "2a505c48-0f05-449a-bce4-570c9e6f82cd"). InnerVolumeSpecName "kube-api-access-69b24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.465348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.465321    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-805edc-29889960-t9ppx"]
Oct 30 22:00:20.571681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.567257    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"]
Oct 30 22:00:20.573020 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.572986    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khd2r\" (UniqueName: \"kubernetes.io/projected/f0df621f-cede-41ef-b8d0-b895f48034d5-kube-api-access-khd2r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.573190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.573175    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69b24\" (UniqueName: \"kubernetes.io/projected/2a505c48-0f05-449a-bce4-570c9e6f82cd-kube-api-access-69b24\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.573298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.573274    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a505c48-0f05-449a-bce4-570c9e6f82cd-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.573298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.573298    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6g6n\" (UniqueName: \"kubernetes.io/projected/d381869f-da79-449f-8a22-1970a5ef7dfa-kube-api-access-d6g6n\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.573298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.573311    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d381869f-da79-449f-8a22-1970a5ef7dfa-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.575215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.575192    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9722ac-29889960-j69jm"]
Oct 30 22:00:20.590935 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.590905    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:20.674408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.674319    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca\") pod \"a9928370-f1a1-4126-9df2-376c65322ed3\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") "
Oct 30 22:00:20.676414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.676384    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq\" (UniqueName: \"kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq\") pod \"a9928370-f1a1-4126-9df2-376c65322ed3\" (UID: \"a9928370-f1a1-4126-9df2-376c65322ed3\") "
Oct 30 22:00:20.678665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.674990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca" pod "a9928370-f1a1-4126-9df2-376c65322ed3" (UID: "a9928370-f1a1-4126-9df2-376c65322ed3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.679128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.679103    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9928370-f1a1-4126-9df2-376c65322ed3-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.680086 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.680059    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq" pod "a9928370-f1a1-4126-9df2-376c65322ed3" (UID: "a9928370-f1a1-4126-9df2-376c65322ed3"). InnerVolumeSpecName "kube-api-access-ngmfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.781112 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:20.781069    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngmfq\" (UniqueName: \"kubernetes.io/projected/a9928370-f1a1-4126-9df2-376c65322ed3-kube-api-access-ngmfq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.063130 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.063093    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"]
Oct 30 22:00:21.075592 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.075565    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"
Oct 30 22:00:21.075851 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.075564    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx" event={"ID":"2a505c48-0f05-449a-bce4-570c9e6f82cd","Type":"ContainerDied","Data":"af2cf397aefbdf1d70de7d0f72db1de09184bf6b47fe1ea604df3cf8743f6c1d"}
Oct 30 22:00:21.075851 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.075758    6695 scope.go:122] "RemoveContainer" containerID="e78a1b66911454a6de3a7df8145d417999758fb8913d16483ccd1bbd0f521e2d"
Oct 30 22:00:21.080888 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.080844    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b" event={"ID":"d381869f-da79-449f-8a22-1970a5ef7dfa","Type":"ContainerDied","Data":"1c73cfd415a0361e9416735a8e21ec47f03a6311085f62612d7b4d05f0aee1e1"}
Oct 30 22:00:21.081024 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.080990    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"
Oct 30 22:00:21.106565 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.106538    6695 scope.go:122] "RemoveContainer" containerID="7dc53d3a1bc4c10850b99640e9fb1bc208b0a421700a359e0fbb9fd3ee577067"
Oct 30 22:00:21.111217 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.111185    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8" event={"ID":"a9928370-f1a1-4126-9df2-376c65322ed3","Type":"ContainerDied","Data":"f004f9b79a58c5e2267ab1f6929f2adbb90a7d5f44384e8bba1a9bc496e9a33e"}
Oct 30 22:00:21.111456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.111437    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"
Oct 30 22:00:21.113591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.113562    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22" event={"ID":"f0df621f-cede-41ef-b8d0-b895f48034d5","Type":"ContainerDied","Data":"18a6e7d0392a8fbd4cb420cef6a7e28f302d6ccfd5b08194dfe737a9266d498f"}
Oct 30 22:00:21.113741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.113707    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"
Oct 30 22:00:21.124843 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.124820    6695 scope.go:122] "RemoveContainer" containerID="691d95016be5c5ba04cc05a136df8e7b9de7d4af44abfd3b20738e992b2643dd"
Oct 30 22:00:21.126643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.126399    6695 generic.go:356] "Generic (PLEG): container finished" podID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerID="7ad857956073dd5999cb7d1bfede5ea900fbb0358cea7687829531bfc68378d1" exitCode=60
Oct 30 22:00:21.126643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.126495    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" event={"ID":"8193d2c6-b8f1-4460-adc6-5eefdda788d9","Type":"ContainerDied","Data":"7ad857956073dd5999cb7d1bfede5ea900fbb0358cea7687829531bfc68378d1"}
Oct 30 22:00:21.127124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.127105    6695 scope.go:122] "RemoveContainer" containerID="7ad857956073dd5999cb7d1bfede5ea900fbb0358cea7687829531bfc68378d1"
Oct 30 22:00:21.128388 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:21.128281    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j_odh-ai-gateway-infra(8193d2c6-b8f1-4460-adc6-5eefdda788d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9"
Oct 30 22:00:21.138734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.138702    6695 generic.go:356] "Generic (PLEG): container finished" podID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerID="1152a34471ac3a4aad40dd2a3bbbca092eb9d0b6b4381fc98774d238ec17fbf4" exitCode=60
Oct 30 22:00:21.138849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.138793    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" event={"ID":"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc","Type":"ContainerDied","Data":"1152a34471ac3a4aad40dd2a3bbbca092eb9d0b6b4381fc98774d238ec17fbf4"}
Oct 30 22:00:21.139317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.139291    6695 scope.go:122] "RemoveContainer" containerID="1152a34471ac3a4aad40dd2a3bbbca092eb9d0b6b4381fc98774d238ec17fbf4"
Oct 30 22:00:21.139878 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:21.139843    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v_odh-ai-gateway-infra(d7e1398e-f2fa-4b74-8525-a7e2017f2fcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc"
Oct 30 22:00:21.143450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.143431    6695 scope.go:122] "RemoveContainer" containerID="865eb7e0ea68ffc5d62b8cd3b42fff3ef1612de74c546f23789971b05acaee2e"
Oct 30 22:00:21.162575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.162551    6695 scope.go:122] "RemoveContainer" containerID="99d4c4cb2e66bfc77b27b0111eadf9acd55248ba4de39a2051cf2bddd47a987f"
Oct 30 22:00:21.213743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.213714    6695 scope.go:122] "RemoveContainer" containerID="c48d17d2f7ad4f552df03403e110a7a79d22af72fc730fedfd7c6a3c3eacf1e3"
Oct 30 22:00:21.286720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.286682    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" path="/var/lib/kubelet/pods/2b2b3430-038d-4ad5-9766-a9c51e1e7046/volumes"
Oct 30 22:00:21.287703 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.287678    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377405e1-8481-444f-a61a-5e59dcf38460" path="/var/lib/kubelet/pods/377405e1-8481-444f-a61a-5e59dcf38460/volumes"
Oct 30 22:00:21.288533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.288498    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e81593-4356-4296-9736-9dae8c77fb77" path="/var/lib/kubelet/pods/51e81593-4356-4296-9736-9dae8c77fb77/volumes"
Oct 30 22:00:21.291166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.291144    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" path="/var/lib/kubelet/pods/916d8393-b1f4-4a2c-a588-6d2f6e86cff8/volumes"
Oct 30 22:00:21.293127 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.293103    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8257c09-556f-4890-a175-529173eaf1a8" path="/var/lib/kubelet/pods/c8257c09-556f-4890-a175-529173eaf1a8/volumes"
Oct 30 22:00:21.294283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.294261    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" path="/var/lib/kubelet/pods/ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f/volumes"
Oct 30 22:00:21.295336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.295315    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"]
Oct 30 22:00:21.303715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.300918    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-tqlnx"]
Oct 30 22:00:21.359973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.359937    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"]
Oct 30 22:00:21.438822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.438547    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"]
Oct 30 22:00:21.446505 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.446476    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19461-29889960-x5rx8"]
Oct 30 22:00:21.482643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.482614    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"]
Oct 30 22:00:21.498806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.494236    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5494e8-29889960-ddj6b"]
Oct 30 22:00:21.498806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.496770    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:21.598446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.598414    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn\" (UniqueName: \"kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn\") pod \"3fa2035c-7a77-4bc7-9141-c912cf623256\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") "
Oct 30 22:00:21.598879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.598853    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca\") pod \"3fa2035c-7a77-4bc7-9141-c912cf623256\" (UID: \"3fa2035c-7a77-4bc7-9141-c912cf623256\") "
Oct 30 22:00:21.600478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.600447    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca" pod "3fa2035c-7a77-4bc7-9141-c912cf623256" (UID: "3fa2035c-7a77-4bc7-9141-c912cf623256"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.601894 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.601867    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn" pod "3fa2035c-7a77-4bc7-9141-c912cf623256" (UID: "3fa2035c-7a77-4bc7-9141-c912cf623256"). InnerVolumeSpecName "kube-api-access-rztkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.603973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.603951    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:21.609172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.609147    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:21.615646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.615474    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:21.620996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.620967    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:21.701075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701037    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz\" (UniqueName: \"kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz\") pod \"c94a3b92-eb35-437d-8edb-5683bef4f768\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") "
Oct 30 22:00:21.701230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701182    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca\") pod \"8df3ce61-1578-490a-b58b-3342bc5b1a01\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") "
Oct 30 22:00:21.701270 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701248    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca\") pod \"4ed668a4-c548-4e09-8d1d-09a08c500c37\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") "
Oct 30 22:00:21.701348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701299    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4\" (UniqueName: \"kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4\") pod \"8df3ce61-1578-490a-b58b-3342bc5b1a01\" (UID: \"8df3ce61-1578-490a-b58b-3342bc5b1a01\") "
Oct 30 22:00:21.701431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701414    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca\") pod \"ed66b735-a7c1-470c-a775-11657620f643\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") "
Oct 30 22:00:21.701577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701559    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk\" (UniqueName: \"kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk\") pod \"4ed668a4-c548-4e09-8d1d-09a08c500c37\" (UID: \"4ed668a4-c548-4e09-8d1d-09a08c500c37\") "
Oct 30 22:00:21.701928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701589    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca" pod "8df3ce61-1578-490a-b58b-3342bc5b1a01" (UID: "8df3ce61-1578-490a-b58b-3342bc5b1a01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.701928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701636    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l\" (UniqueName: \"kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l\") pod \"ed66b735-a7c1-470c-a775-11657620f643\" (UID: \"ed66b735-a7c1-470c-a775-11657620f643\") "
Oct 30 22:00:21.701928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701662    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca\") pod \"c94a3b92-eb35-437d-8edb-5683bef4f768\" (UID: \"c94a3b92-eb35-437d-8edb-5683bef4f768\") "
Oct 30 22:00:21.701928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701649    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca" pod "4ed668a4-c548-4e09-8d1d-09a08c500c37" (UID: "4ed668a4-c548-4e09-8d1d-09a08c500c37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.701928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.701835    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca" pod "ed66b735-a7c1-470c-a775-11657620f643" (UID: "ed66b735-a7c1-470c-a775-11657620f643"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.702206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702078    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca" pod "c94a3b92-eb35-437d-8edb-5683bef4f768" (UID: "c94a3b92-eb35-437d-8edb-5683bef4f768"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.702394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702375    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztkn\" (UniqueName: \"kubernetes.io/projected/3fa2035c-7a77-4bc7-9141-c912cf623256-kube-api-access-rztkn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.702482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702400    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66b735-a7c1-470c-a775-11657620f643-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.702482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702414    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa2035c-7a77-4bc7-9141-c912cf623256-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.702482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702428    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94a3b92-eb35-437d-8edb-5683bef4f768-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.702482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702441    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8df3ce61-1578-490a-b58b-3342bc5b1a01-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.702482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.702456    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed668a4-c548-4e09-8d1d-09a08c500c37-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.703357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.703337    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz" pod "c94a3b92-eb35-437d-8edb-5683bef4f768" (UID: "c94a3b92-eb35-437d-8edb-5683bef4f768"). InnerVolumeSpecName "kube-api-access-4fgtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.703848 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.703830    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk" pod "4ed668a4-c548-4e09-8d1d-09a08c500c37" (UID: "4ed668a4-c548-4e09-8d1d-09a08c500c37"). InnerVolumeSpecName "kube-api-access-hnzgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.704145 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.704105    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l" pod "ed66b735-a7c1-470c-a775-11657620f643" (UID: "ed66b735-a7c1-470c-a775-11657620f643"). InnerVolumeSpecName "kube-api-access-r799l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.704251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.704147    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4" pod "8df3ce61-1578-490a-b58b-3342bc5b1a01" (UID: "8df3ce61-1578-490a-b58b-3342bc5b1a01"). InnerVolumeSpecName "kube-api-access-tk9h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.709730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.709708    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"]
Oct 30 22:00:21.714060 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.714039    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-e06b90-29889960-mvn22"]
Oct 30 22:00:21.803905 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.803867    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzgk\" (UniqueName: \"kubernetes.io/projected/4ed668a4-c548-4e09-8d1d-09a08c500c37-kube-api-access-hnzgk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.804104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.803907    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r799l\" (UniqueName: \"kubernetes.io/projected/ed66b735-a7c1-470c-a775-11657620f643-kube-api-access-r799l\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.804104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.803926    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgtz\" (UniqueName: \"kubernetes.io/projected/c94a3b92-eb35-437d-8edb-5683bef4f768-kube-api-access-4fgtz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.804104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:21.803940    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9h4\" (UniqueName: \"kubernetes.io/projected/8df3ce61-1578-490a-b58b-3342bc5b1a01-kube-api-access-tk9h4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:22.179270 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.179236    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"
Oct 30 22:00:22.179826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.179242    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc" event={"ID":"8df3ce61-1578-490a-b58b-3342bc5b1a01","Type":"ContainerDied","Data":"5f9c53b248d68b872bbe805668ddf53b855ad63d2bfd135b7dbdce1c924416fe"}
Oct 30 22:00:22.179826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.179409    6695 scope.go:122] "RemoveContainer" containerID="c09afb93c86ecabaff9173fc930b5f61c66b65fe28e810d588cec8764125cbb5"
Oct 30 22:00:22.180756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.180727    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk" event={"ID":"4ed668a4-c548-4e09-8d1d-09a08c500c37","Type":"ContainerDied","Data":"66c6360050922e774109c0931ac580a65d183fe74e206886cfafcc632bc9e03b"}
Oct 30 22:00:22.180850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.180740    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"
Oct 30 22:00:22.182593 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.182573    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"
Oct 30 22:00:22.183036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.182592    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g" event={"ID":"c94a3b92-eb35-437d-8edb-5683bef4f768","Type":"ContainerDied","Data":"80bca83916b5239c151bebb276257cccd87d5c901ee649b45357cbd4c169e909"}
Oct 30 22:00:22.184426 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.184408    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"
Oct 30 22:00:22.184536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.184464    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f" event={"ID":"ed66b735-a7c1-470c-a775-11657620f643","Type":"ContainerDied","Data":"6f4225e002c5a451d2f5f97ede57a7081eca6f622a3e2b8484aaa7975977065b"}
Oct 30 22:00:22.191531 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.190874    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"
Oct 30 22:00:22.191531 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.190897    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r" event={"ID":"3fa2035c-7a77-4bc7-9141-c912cf623256","Type":"ContainerDied","Data":"21da69969c3762218800957538ece66adf6e7f0de0d260fc8de80b6015ba4af3"}
Oct 30 22:00:22.198761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.197908    6695 scope.go:122] "RemoveContainer" containerID="876bf6365d6418ed87fe2e11d8b7e01580b950dcf0d3222b96dbcd70cfd53eb5"
Oct 30 22:00:22.286669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.286643    6695 scope.go:122] "RemoveContainer" containerID="9f52d357c3881ad1a9d240e3ba6364f642d9dabcdcca1d81f68840dbf6e9a738"
Oct 30 22:00:22.329660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.328967    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"]
Oct 30 22:00:22.335764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.335736    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f49db2-29889960-fzj7r"]
Oct 30 22:00:22.364762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.364727    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"]
Oct 30 22:00:22.366213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.366125    6695 scope.go:122] "RemoveContainer" containerID="2f4712f43ce17c93542289e7bab24eb516f591f3fd349887b56d3c14ab7c64c4"
Oct 30 22:00:22.397722 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.394030    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ea40ee-29889960-vq29g"]
Oct 30 22:00:22.417319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.416969    6695 scope.go:122] "RemoveContainer" containerID="de727a0ff06c87b9ab217a7fbf8af76f34122bab50ca408762294378eaab166b"
Oct 30 22:00:22.426335 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.426205    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"]
Oct 30 22:00:22.441200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.440976    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"]
Oct 30 22:00:22.445734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.445703    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ed0866-29889960-7zb4f"]
Oct 30 22:00:22.477051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.476992    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"]
Oct 30 22:00:22.483810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.483757    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3d5d13-29889960-v79nk"]
Oct 30 22:00:22.501471 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.501411    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"]
Oct 30 22:00:22.508178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.508143    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5d3601-29889960-7gkcc"]
Oct 30 22:00:22.601218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.601181    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"]
Oct 30 22:00:22.623690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.623665    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:22.738107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.738031    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs\" (UniqueName: \"kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs\") pod \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") "
Oct 30 22:00:22.738589 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.738562    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca\") pod \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\" (UID: \"8193d2c6-b8f1-4460-adc6-5eefdda788d9\") "
Oct 30 22:00:22.739272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.739249    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca" pod "8193d2c6-b8f1-4460-adc6-5eefdda788d9" (UID: "8193d2c6-b8f1-4460-adc6-5eefdda788d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:22.741277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.741246    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs" pod "8193d2c6-b8f1-4460-adc6-5eefdda788d9" (UID: "8193d2c6-b8f1-4460-adc6-5eefdda788d9"). InnerVolumeSpecName "kube-api-access-qwzjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:22.776907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.776884    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:22.863707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.848963    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca\") pod \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") "
Oct 30 22:00:22.863707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.849172    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7\" (UniqueName: \"kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7\") pod \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\" (UID: \"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc\") "
Oct 30 22:00:22.863707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.850070    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8193d2c6-b8f1-4460-adc6-5eefdda788d9-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:22.863707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.850093    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwzjs\" (UniqueName: \"kubernetes.io/projected/8193d2c6-b8f1-4460-adc6-5eefdda788d9-kube-api-access-qwzjs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:22.863707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.855920    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca" pod "d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" (UID: "d7e1398e-f2fa-4b74-8525-a7e2017f2fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:22.903989 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.903944    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7" pod "d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" (UID: "d7e1398e-f2fa-4b74-8525-a7e2017f2fcc"). InnerVolumeSpecName "kube-api-access-ccfz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:22.963544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.963509    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:22.963544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:22.963540    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccfz7\" (UniqueName: \"kubernetes.io/projected/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc-kube-api-access-ccfz7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:23.063243 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.063201    6695 scope.go:122] "RemoveContainer" containerID="117b32de4501f3b36920692bac35a16e3a4746787f71843b68069ba781a5a1c9"
Oct 30 22:00:23.074853 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.074830    6695 scope.go:122] "RemoveContainer" containerID="fb7ec23fb895bd9d7984ea4e48e3692a2e435cfbd64bc2aeda5592fae7cd80f5"
Oct 30 22:00:23.085500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.085478    6695 scope.go:122] "RemoveContainer" containerID="14683120529b77122bd39fe7ec1971ac78e0679dd09f73c5ba0e24d3ecb6f795"
Oct 30 22:00:23.217935 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.217912    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"
Oct 30 22:00:23.219687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.219664    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"
Oct 30 22:00:23.220344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.220303    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" path="/var/lib/kubelet/pods/2a505c48-0f05-449a-bce4-570c9e6f82cd/volumes"
Oct 30 22:00:23.220850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.220832    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" path="/var/lib/kubelet/pods/3fa2035c-7a77-4bc7-9141-c912cf623256/volumes"
Oct 30 22:00:23.221295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.221280    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" path="/var/lib/kubelet/pods/4ed668a4-c548-4e09-8d1d-09a08c500c37/volumes"
Oct 30 22:00:23.222065 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.222049    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" path="/var/lib/kubelet/pods/8df3ce61-1578-490a-b58b-3342bc5b1a01/volumes"
Oct 30 22:00:23.222553 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.222534    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" path="/var/lib/kubelet/pods/a9928370-f1a1-4126-9df2-376c65322ed3/volumes"
Oct 30 22:00:23.223173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.223150    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" path="/var/lib/kubelet/pods/c94a3b92-eb35-437d-8edb-5683bef4f768/volumes"
Oct 30 22:00:23.224168 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.224124    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" path="/var/lib/kubelet/pods/d381869f-da79-449f-8a22-1970a5ef7dfa/volumes"
Oct 30 22:00:23.224794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.224689    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed66b735-a7c1-470c-a775-11657620f643" path="/var/lib/kubelet/pods/ed66b735-a7c1-470c-a775-11657620f643/volumes"
Oct 30 22:00:23.225270 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.225249    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" path="/var/lib/kubelet/pods/f0df621f-cede-41ef-b8d0-b895f48034d5/volumes"
Oct 30 22:00:23.226806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.226581    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j" event={"ID":"8193d2c6-b8f1-4460-adc6-5eefdda788d9","Type":"ContainerDied","Data":"129c4e27b3a8fb3ec1f7a3f88e5a586de6519265f9aa95dc4f2fff47ee48cc6c"}
Oct 30 22:00:23.226806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.226635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v" event={"ID":"d7e1398e-f2fa-4b74-8525-a7e2017f2fcc","Type":"ContainerDied","Data":"03dc0e2f3b108f3837e10b55982789762af780724168611cb077f6b683f89bc7"}
Oct 30 22:00:23.226806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.226672    6695 scope.go:122] "RemoveContainer" containerID="7ad857956073dd5999cb7d1bfede5ea900fbb0358cea7687829531bfc68378d1"
Oct 30 22:00:23.265709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.265624    6695 scope.go:122] "RemoveContainer" containerID="1152a34471ac3a4aad40dd2a3bbbca092eb9d0b6b4381fc98774d238ec17fbf4"
Oct 30 22:00:23.292812 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.292784    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"]
Oct 30 22:00:23.298401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.298310    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846656-29889960-7hr9j"]
Oct 30 22:00:23.333025 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.332992    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"]
Oct 30 22:00:23.337493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:23.337461    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-427a6e-29889960-7ct7v"]
Oct 30 22:00:25.280739 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:25.280697    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" path="/var/lib/kubelet/pods/8193d2c6-b8f1-4460-adc6-5eefdda788d9/volumes"
Oct 30 22:00:25.281564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:25.281539    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" path="/var/lib/kubelet/pods/d7e1398e-f2fa-4b74-8525-a7e2017f2fcc/volumes"
Oct 30 22:00:26.871251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.871214    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"]
Oct 30 22:00:26.872017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872001    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872018    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872029    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872034    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872047    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872053    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872062    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872067    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872073    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:26.872075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872078    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872084    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872089    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872099    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872104    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872111    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872115    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872124    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872128    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872135    6695 cpu_manager.go:414] "containerMap: removing container" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872140    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872151    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872158    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872169    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872176    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872184    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872190    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872198    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872205    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872213    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872219    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872236    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872243    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872257    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872264    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872278    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872285    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872294    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872300    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872314    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872320    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872345    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872352    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872362    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5619f6cb-592c-47c1-9bfa-d9f10ff261a0" containerName="revoke-keys"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872370    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5619f6cb-592c-47c1-9bfa-d9f10ff261a0" containerName="revoke-keys"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872381    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872390    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872400    6695 cpu_manager.go:414] "containerMap: removing container" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.872390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872409    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872417    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872424    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872440    6695 cpu_manager.go:414] "containerMap: removing container" podUID="51e81593-4356-4296-9736-9dae8c77fb77" containerName="payload-processing"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872447    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e81593-4356-4296-9736-9dae8c77fb77" containerName="payload-processing"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872458    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872465    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872479    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872485    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872499    6695 cpu_manager.go:414] "containerMap: removing container" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872505    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872512    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872517    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872524    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872528    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872537    6695 cpu_manager.go:414] "containerMap: removing container" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872541    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872554    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872558    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872567    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872571    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872579    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872582    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872591    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872595    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872624    6695 cpu_manager.go:414] "containerMap: removing container" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872632    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872643    6695 cpu_manager.go:414] "containerMap: removing container" podUID="dd615063-3ebf-4b00-ace6-0ee6f5a0076a" containerName="payload-processing"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872648    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd615063-3ebf-4b00-ace6-0ee6f5a0076a" containerName="payload-processing"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872663    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872670    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872681    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872685    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872872    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872881    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872888    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872895    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872900    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872907    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872912    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872919    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872924    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872930    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd615063-3ebf-4b00-ace6-0ee6f5a0076a" containerName="payload-processing"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872936    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872941    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872947    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872952    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872958    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872965    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872970    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872976    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872980    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872986    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872992    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a505c48-0f05-449a-bce4-570c9e6f82cd" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.872999    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873005    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873010    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873015    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873019    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873025    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873033    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873039    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873045    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa2035c-7a77-4bc7-9141-c912cf623256" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873050    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5619f6cb-592c-47c1-9bfa-d9f10ff261a0" containerName="revoke-keys"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873059    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873065    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873072    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873078    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e81593-4356-4296-9736-9dae8c77fb77" containerName="payload-processing"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873086    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873093    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed66b735-a7c1-470c-a775-11657620f643" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873097    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873103    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873118    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:26.874163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.873125    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="377405e1-8481-444f-a61a-5e59dcf38460" containerName="cleanup"
Oct 30 22:00:26.880975 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.880956    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:26.896283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.896256    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"]
Oct 30 22:00:26.901798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.901768    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-plugins-e2e-extmod-b-846656\""
Oct 30 22:00:26.901947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.901924    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"openshift-service-ca.crt\""
Oct 30 22:00:26.901947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.901948    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"routing-overlay\""
Oct 30 22:00:26.902128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.901967    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-f060d103b140\""
Oct 30 22:00:26.902128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.902040    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"kube-root-ca.crt\""
Oct 30 22:00:26.909094 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.909070    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-e2e--f95d90a5-dockercfg-5z6mv\""
Oct 30 22:00:26.970547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.970509    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:26.970914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.970879    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:26.971221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.971190    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh42d\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:26.971397 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:26.971342    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.082629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.080327    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh42d\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.082629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.080442    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.082629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.080512    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.082629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.080583    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.082629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.081554    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.082629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.082390    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.097641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.092360    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.114617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.114567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh42d\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d\") pod \"payload-processing-external-model-e2e-extmod-b-846656-75665h9fk\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.199015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.198923    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:27.524217 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.524143    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"]
Oct 30 22:00:27.825510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.825427    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"]
Oct 30 22:00:27.826336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826317    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:27.826336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826337    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826363    6695 cpu_manager.go:414] "containerMap: removing container" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826368    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826390    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826396    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826410    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826415    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826424    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826429    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826447    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:27.826461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826454    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94a3b92-eb35-437d-8edb-5683bef4f768" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826467    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826471    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826483    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826488    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9928370-f1a1-4126-9df2-376c65322ed3" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826494    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826499    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826508    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826512    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826523    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826529    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826653    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0df621f-cede-41ef-b8d0-b895f48034d5" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826663    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed668a4-c548-4e09-8d1d-09a08c500c37" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826669    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df3ce61-1578-490a-b58b-3342bc5b1a01" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826677    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2542d6-ffac-4d16-9cb0-f6f9ab97e51f" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826685    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2b3430-038d-4ad5-9766-a9c51e1e7046" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826697    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8193d2c6-b8f1-4460-adc6-5eefdda788d9" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826708    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8257c09-556f-4890-a175-529173eaf1a8" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826714    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e1398e-f2fa-4b74-8525-a7e2017f2fcc" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826723    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="916d8393-b1f4-4a2c-a588-6d2f6e86cff8" containerName="cleanup"
Oct 30 22:00:27.826813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.826731    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d381869f-da79-449f-8a22-1970a5ef7dfa" containerName="cleanup"
Oct 30 22:00:27.830867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.830849    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:27.845185 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.845156    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 22:00:27.849799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.849770    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"]
Oct 30 22:00:27.854794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.854767    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-010bb35e298d\""
Oct 30 22:00:27.854991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.854788    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 22:00:27.854991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.854871    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 22:00:27.854991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.854920    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 22:00:27.855242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.854922    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-pxsz6\""
Oct 30 22:00:27.895788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.895744    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:27.896298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.895800    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8m6l\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:27.896298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.895972    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:27.896298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:27.896071    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.012802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.012756    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.030730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.030698    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8m6l\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.031413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.031387    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.032075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.032051    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.033125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.030575    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.033285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.032813    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.045287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.045259    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.060259 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.060196    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8m6l\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.148495 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.148079    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:28.327585 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.327543    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk" event={"ID":"41951b3c-2e76-4baa-8d0a-2febd06723e1","Type":"ContainerStarted","Data":"5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35"}
Oct 30 22:00:28.327831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.327640    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk" event={"ID":"41951b3c-2e76-4baa-8d0a-2febd06723e1","Type":"ContainerStarted","Data":"45d8e55ede13e0b2f7d03760f9d635629e0c812612bc714c1d6206ad190ca01e"}
Oct 30 22:00:28.327831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.327685    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:28.375461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.375407    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk" podStartSLOduration=2.375388944 podStartE2EDuration="2.375388944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:28.367697588 +0000 UTC m=+3011.290155887" watchObservedRunningTime="2026-10-30 22:00:28.375388944 +0000 UTC m=+3011.297847243"
Oct 30 22:00:28.488394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:28.488296    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"]
Oct 30 22:00:28.494243 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:28.494215    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e5b0c6_0329_4692_ac40_6f993cb8abc0.slice/crio-35760bd23ba3dacbdf3c5efeb71a6bc221dd85f6e9db26339dfa2d8e41c1059f WatchSource:0}: Error finding container 35760bd23ba3dacbdf3c5efeb71a6bc221dd85f6e9db26339dfa2d8e41c1059f: Status 404 returned error can't find the container with id 35760bd23ba3dacbdf3c5efeb71a6bc221dd85f6e9db26339dfa2d8e41c1059f
Oct 30 22:00:29.337300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:29.337258    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6" event={"ID":"48e5b0c6-0329-4692-ac40-6f993cb8abc0","Type":"ContainerStarted","Data":"6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35"}
Oct 30 22:00:29.337748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:29.337325    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6" event={"ID":"48e5b0c6-0329-4692-ac40-6f993cb8abc0","Type":"ContainerStarted","Data":"35760bd23ba3dacbdf3c5efeb71a6bc221dd85f6e9db26339dfa2d8e41c1059f"}
Oct 30 22:00:29.337748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:29.337505    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:29.378692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:29.378597    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6" podStartSLOduration=2.378581471 podStartE2EDuration="2.378581471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:29.375218279 +0000 UTC m=+3012.297676576" watchObservedRunningTime="2026-10-30 22:00:29.378581471 +0000 UTC m=+3012.301039772"
Oct 30 22:00:31.074503 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:31.074464    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:00:32.422625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:32.421080    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"]
Oct 30 22:00:32.422625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:32.421352    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" containerID="cri-o://cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316" gracePeriod=30
Oct 30 22:00:34.491585 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:34.491551    6695 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:00:34.492076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:34.491664    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:00:37.671735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.671708    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 22:00:37.798093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798053    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798165    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798193    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798277    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798508 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798335    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798508 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798390    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798508 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798473    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798710 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798516    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.798769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.798722    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs\") pod \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\" (UID: \"1cb42d7c-2086-4352-a6b3-3e8f3c09da77\") "
Oct 30 22:00:37.799155 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.799125    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:37.799326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.799196    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:37.799449 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.799237    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:37.799547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.799443    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.799912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.799575    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:37.799912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.799749    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:37.801671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.801640    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:37.801977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.801949    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:37.802062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.801959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:37.802504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.802448    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk" pod "1cb42d7c-2086-4352-a6b3-3e8f3c09da77" (UID: "1cb42d7c-2086-4352-a6b3-3e8f3c09da77"). InnerVolumeSpecName "kube-api-access-dj7pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:37.901107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901071    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901103    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901112    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7pk\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-kube-api-access-dj7pk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901123    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901132    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901139    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901148    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:37.901375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:37.901155    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42d7c-2086-4352-a6b3-3e8f3c09da77-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:38.491032 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.490999    6695 generic.go:356] "Generic (PLEG): container finished" podID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerID="cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316" exitCode=0
Oct 30 22:00:38.491225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.491047    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"
Oct 30 22:00:38.491225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.491090    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" event={"ID":"1cb42d7c-2086-4352-a6b3-3e8f3c09da77","Type":"ContainerDied","Data":"cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316"}
Oct 30 22:00:38.491225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.491149    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj" event={"ID":"1cb42d7c-2086-4352-a6b3-3e8f3c09da77","Type":"ContainerDied","Data":"bd2befcad7ae98ec263ab6b37c1a23d4aa3109c075358ddcc44e79d459ed7402"}
Oct 30 22:00:38.491225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.491173    6695 scope.go:122] "RemoveContainer" containerID="cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316"
Oct 30 22:00:38.504906 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.504885    6695 scope.go:122] "RemoveContainer" containerID="e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"
Oct 30 22:00:38.516799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.516769    6695 scope.go:122] "RemoveContainer" containerID="cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316"
Oct 30 22:00:38.517135 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:38.517113    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316\": container with ID starting with cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316 not found: ID does not exist" containerID="cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316"
Oct 30 22:00:38.517234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.517143    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316"} err="failed to get container status \"cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316\": rpc error: code = NotFound desc = could not find container \"cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316\": container with ID starting with cc733972d85660c27ad4a64ffd246c0467eb98f54281901a9da8ac111d0ae316 not found: ID does not exist"
Oct 30 22:00:38.517234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.517160    6695 scope.go:122] "RemoveContainer" containerID="e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"
Oct 30 22:00:38.517493 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:38.517471    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6\": container with ID starting with e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6 not found: ID does not exist" containerID="e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"
Oct 30 22:00:38.517563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.517499    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6"} err="failed to get container status \"e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6\": rpc error: code = NotFound desc = could not find container \"e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6\": container with ID starting with e05d2362ca084077939f87a8d582022a4293b91d546306c38947fd1b872fdee6 not found: ID does not exist"
Oct 30 22:00:38.588579 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.588087    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"]
Oct 30 22:00:38.588579 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:38.588159    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-7nkzj"]
Oct 30 22:00:39.216971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:39.216938    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" path="/var/lib/kubelet/pods/1cb42d7c-2086-4352-a6b3-3e8f3c09da77/volumes"
Oct 30 22:00:39.340045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:39.340013    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:40.348344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:40.348316    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:46.233652 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:46.233592    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"]
Oct 30 22:00:47.030359 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:47.030313    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"]
Oct 30 22:00:47.041244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:47.041207    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae23572b-bq2m7"]
Oct 30 22:00:47.217361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:47.217327    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae41f69-7450-4b7c-b65a-fe872e9bd4ac" path="/var/lib/kubelet/pods/bae41f69-7450-4b7c-b65a-fe872e9bd4ac/volumes"
Oct 30 22:00:47.242616 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:47.242574    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"]
Oct 30 22:00:51.234431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.234377    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk" podUID="41951b3c-2e76-4baa-8d0a-2febd06723e1" containerName="payload-processing" containerID="cri-o://5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35" gracePeriod=25
Oct 30 22:00:51.593856 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.593817    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:51.622969 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.622933    6695 generic.go:356] "Generic (PLEG): container finished" podID="41951b3c-2e76-4baa-8d0a-2febd06723e1" containerID="5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35" exitCode=0
Oct 30 22:00:51.623157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.622976    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"
Oct 30 22:00:51.623157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.623009    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk" event={"ID":"41951b3c-2e76-4baa-8d0a-2febd06723e1","Type":"ContainerDied","Data":"5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35"}
Oct 30 22:00:51.623157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.623062    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk" event={"ID":"41951b3c-2e76-4baa-8d0a-2febd06723e1","Type":"ContainerDied","Data":"45d8e55ede13e0b2f7d03760f9d635629e0c812612bc714c1d6206ad190ca01e"}
Oct 30 22:00:51.623157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.623084    6695 scope.go:122] "RemoveContainer" containerID="5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35"
Oct 30 22:00:51.635221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.635200    6695 scope.go:122] "RemoveContainer" containerID="5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35"
Oct 30 22:00:51.635555 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:51.635532    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35\": container with ID starting with 5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35 not found: ID does not exist" containerID="5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35"
Oct 30 22:00:51.635691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.635563    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35"} err="failed to get container status \"5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35\": rpc error: code = NotFound desc = could not find container \"5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35\": container with ID starting with 5a39420ce1e811306d470b904b2be382adcb46a057bbf4e6342307d500f33f35 not found: ID does not exist"
Oct 30 22:00:51.761000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.760968    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials\") pod \"41951b3c-2e76-4baa-8d0a-2febd06723e1\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") "
Oct 30 22:00:51.761197 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761041    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume\") pod \"41951b3c-2e76-4baa-8d0a-2febd06723e1\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") "
Oct 30 22:00:51.761269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761201    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume\") pod \"41951b3c-2e76-4baa-8d0a-2febd06723e1\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") "
Oct 30 22:00:51.761269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761249    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d\") pod \"41951b3c-2e76-4baa-8d0a-2febd06723e1\" (UID: \"41951b3c-2e76-4baa-8d0a-2febd06723e1\") "
Oct 30 22:00:51.761480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761456    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume" pod "41951b3c-2e76-4baa-8d0a-2febd06723e1" (UID: "41951b3c-2e76-4baa-8d0a-2febd06723e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:51.761645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761622    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume" pod "41951b3c-2e76-4baa-8d0a-2febd06723e1" (UID: "41951b3c-2e76-4baa-8d0a-2febd06723e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:51.761960 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761942    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:51.762038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.761964    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41951b3c-2e76-4baa-8d0a-2febd06723e1-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:51.763177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.763154    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials" pod "41951b3c-2e76-4baa-8d0a-2febd06723e1" (UID: "41951b3c-2e76-4baa-8d0a-2febd06723e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:51.763289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.763273    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d" pod "41951b3c-2e76-4baa-8d0a-2febd06723e1" (UID: "41951b3c-2e76-4baa-8d0a-2febd06723e1"). InnerVolumeSpecName "kube-api-access-wh42d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:51.863151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.863054    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh42d\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-kube-api-access-wh42d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:51.863151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.863089    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41951b3c-2e76-4baa-8d0a-2febd06723e1-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:51.970342 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.970305    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"]
Oct 30 22:00:51.976917 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:51.976886    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-75665h9fk"]
Oct 30 22:00:52.243197 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.243084    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6" podUID="48e5b0c6-0329-4692-ac40-6f993cb8abc0" containerName="payload-processing" containerID="cri-o://6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35" gracePeriod=25
Oct 30 22:00:52.532553 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.532526    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:52.634699 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.634654    6695 generic.go:356] "Generic (PLEG): container finished" podID="48e5b0c6-0329-4692-ac40-6f993cb8abc0" containerID="6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35" exitCode=0
Oct 30 22:00:52.634939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.634731    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"
Oct 30 22:00:52.635030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.635007    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6" event={"ID":"48e5b0c6-0329-4692-ac40-6f993cb8abc0","Type":"ContainerDied","Data":"6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35"}
Oct 30 22:00:52.635121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.635102    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6" event={"ID":"48e5b0c6-0329-4692-ac40-6f993cb8abc0","Type":"ContainerDied","Data":"35760bd23ba3dacbdf3c5efeb71a6bc221dd85f6e9db26339dfa2d8e41c1059f"}
Oct 30 22:00:52.635192 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.635123    6695 scope.go:122] "RemoveContainer" containerID="6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35"
Oct 30 22:00:52.648863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.648844    6695 scope.go:122] "RemoveContainer" containerID="6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35"
Oct 30 22:00:52.649222 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:00:52.649185    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35\": container with ID starting with 6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35 not found: ID does not exist" containerID="6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35"
Oct 30 22:00:52.649319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.649232    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35"} err="failed to get container status \"6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35\": rpc error: code = NotFound desc = could not find container \"6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35\": container with ID starting with 6611e1f9e55e34aa0f2308dd9f1364d820dd5f75ea9ebe4b592bffdf2f198c35 not found: ID does not exist"
Oct 30 22:00:52.678201 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.678173    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume\") pod \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") "
Oct 30 22:00:52.678370 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.678356    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials\") pod \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") "
Oct 30 22:00:52.678487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.678473    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l\") pod \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") "
Oct 30 22:00:52.678559 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.678534    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume" pod "48e5b0c6-0329-4692-ac40-6f993cb8abc0" (UID: "48e5b0c6-0329-4692-ac40-6f993cb8abc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:52.678658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.678643    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume\") pod \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\" (UID: \"48e5b0c6-0329-4692-ac40-6f993cb8abc0\") "
Oct 30 22:00:52.679014 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.678992    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume" pod "48e5b0c6-0329-4692-ac40-6f993cb8abc0" (UID: "48e5b0c6-0329-4692-ac40-6f993cb8abc0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:52.681586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.680305    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:52.681586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.680342    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48e5b0c6-0329-4692-ac40-6f993cb8abc0-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:52.687394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.686514    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials" pod "48e5b0c6-0329-4692-ac40-6f993cb8abc0" (UID: "48e5b0c6-0329-4692-ac40-6f993cb8abc0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:52.687394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.686867    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l" pod "48e5b0c6-0329-4692-ac40-6f993cb8abc0" (UID: "48e5b0c6-0329-4692-ac40-6f993cb8abc0"). InnerVolumeSpecName "kube-api-access-v8m6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:52.782303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.782256    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:52.782303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.782293    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8m6l\" (UniqueName: \"kubernetes.io/projected/48e5b0c6-0329-4692-ac40-6f993cb8abc0-kube-api-access-v8m6l\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:52.982873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.982834    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"]
Oct 30 22:00:52.988683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:52.988654    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f84n9nm6"]
Oct 30 22:00:53.219000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:53.218921    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41951b3c-2e76-4baa-8d0a-2febd06723e1" path="/var/lib/kubelet/pods/41951b3c-2e76-4baa-8d0a-2febd06723e1/volumes"
Oct 30 22:00:53.219468 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:53.219451    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e5b0c6-0329-4692-ac40-6f993cb8abc0" path="/var/lib/kubelet/pods/48e5b0c6-0329-4692-ac40-6f993cb8abc0/volumes"
Oct 30 22:00:55.235931 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.235895    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"]
Oct 30 22:00:55.236739 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236722    6695 cpu_manager.go:414] "containerMap: removing container" podUID="41951b3c-2e76-4baa-8d0a-2febd06723e1" containerName="payload-processing"
Oct 30 22:00:55.236796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236740    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41951b3c-2e76-4baa-8d0a-2febd06723e1" containerName="payload-processing"
Oct 30 22:00:55.236796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236753    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy"
Oct 30 22:00:55.236796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236759    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy"
Oct 30 22:00:55.236796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236768    6695 cpu_manager.go:414] "containerMap: removing container" podUID="48e5b0c6-0329-4692-ac40-6f993cb8abc0" containerName="payload-processing"
Oct 30 22:00:55.236796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236774    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e5b0c6-0329-4692-ac40-6f993cb8abc0" containerName="payload-processing"
Oct 30 22:00:55.236972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236940    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy"
Oct 30 22:00:55.236972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236955    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="41951b3c-2e76-4baa-8d0a-2febd06723e1" containerName="payload-processing"
Oct 30 22:00:55.236972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236961    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy"
Oct 30 22:00:55.236972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.236967    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e5b0c6-0329-4692-ac40-6f993cb8abc0" containerName="payload-processing"
Oct 30 22:00:55.242500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.242477    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.258977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.258955    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:55.259118 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.259104    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:00:55.270265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.270233    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:55.275024 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.274996    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"]
Oct 30 22:00:55.328388 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.328345    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.328569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.328457    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.328729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.328708    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnn6\" (UniqueName: \"kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.358516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.358488    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:55.430420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.430388    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnn6\" (UniqueName: \"kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.430636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.430487    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.430636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.430545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.431225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.431197    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.433101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.433072    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.452475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.452448    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnn6\" (UniqueName: \"kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6\") pod \"maas-api-revoke-sub-07d875d3-8mlgq\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.557198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.557156    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:55.870693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:55.870670    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"]
Oct 30 22:00:55.873334 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:00:55.873304    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d36087_dcb8_4cb8_9c99_baebdc35f03a.slice/crio-74cacac53f3486f2679336bba170dbeb0ec530850b8f3f4296225790948faafa WatchSource:0}: Error finding container 74cacac53f3486f2679336bba170dbeb0ec530850b8f3f4296225790948faafa: Status 404 returned error can't find the container with id 74cacac53f3486f2679336bba170dbeb0ec530850b8f3f4296225790948faafa
Oct 30 22:00:56.675864 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:56.675822    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq" event={"ID":"15d36087-dcb8-4cb8-9c99-baebdc35f03a","Type":"ContainerStarted","Data":"9790c610734f2db96d45d2d6354559c4c21646b9beab0ea7b7158bc634bf6b4e"}
Oct 30 22:00:56.676353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:56.675880    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq" event={"ID":"15d36087-dcb8-4cb8-9c99-baebdc35f03a","Type":"ContainerStarted","Data":"74cacac53f3486f2679336bba170dbeb0ec530850b8f3f4296225790948faafa"}
Oct 30 22:00:56.714528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:56.714476    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq" podStartSLOduration=1.7144630809999999 podStartE2EDuration="1.714463081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:56.713825196 +0000 UTC m=+3039.636283494" watchObservedRunningTime="2026-10-30 22:00:56.714463081 +0000 UTC m=+3039.636921400"
Oct 30 22:00:57.686846 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:57.686813    6695 generic.go:356] "Generic (PLEG): container finished" podID="15d36087-dcb8-4cb8-9c99-baebdc35f03a" containerID="9790c610734f2db96d45d2d6354559c4c21646b9beab0ea7b7158bc634bf6b4e" exitCode=0
Oct 30 22:00:57.687278 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:57.686906    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq" event={"ID":"15d36087-dcb8-4cb8-9c99-baebdc35f03a","Type":"ContainerDied","Data":"9790c610734f2db96d45d2d6354559c4c21646b9beab0ea7b7158bc634bf6b4e"}
Oct 30 22:00:58.868181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.868152    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:00:58.985672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.985567    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6\" (UniqueName: \"kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6\") pod \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") "
Oct 30 22:00:58.985851 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.985716    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca\") pod \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") "
Oct 30 22:00:58.986024 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.986007    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca\") pod \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\" (UID: \"15d36087-dcb8-4cb8-9c99-baebdc35f03a\") "
Oct 30 22:00:58.986493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.986464    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca" pod "15d36087-dcb8-4cb8-9c99-baebdc35f03a" (UID: "15d36087-dcb8-4cb8-9c99-baebdc35f03a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:58.986705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.986683    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:58.987902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.987875    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6" pod "15d36087-dcb8-4cb8-9c99-baebdc35f03a" (UID: "15d36087-dcb8-4cb8-9c99-baebdc35f03a"). InnerVolumeSpecName "kube-api-access-khnn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:58.988116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:58.988098    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca" pod "15d36087-dcb8-4cb8-9c99-baebdc35f03a" (UID: "15d36087-dcb8-4cb8-9c99-baebdc35f03a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:59.088032 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:59.087993    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khnn6\" (UniqueName: \"kubernetes.io/projected/15d36087-dcb8-4cb8-9c99-baebdc35f03a-kube-api-access-khnn6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:59.088032 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:59.088025    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15d36087-dcb8-4cb8-9c99-baebdc35f03a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:00:59.749635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:59.749085    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq" event={"ID":"15d36087-dcb8-4cb8-9c99-baebdc35f03a","Type":"ContainerDied","Data":"74cacac53f3486f2679336bba170dbeb0ec530850b8f3f4296225790948faafa"}
Oct 30 22:00:59.749635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:59.749154    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74cacac53f3486f2679336bba170dbeb0ec530850b8f3f4296225790948faafa"
Oct 30 22:00:59.749635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:00:59.749231    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"
Oct 30 22:01:01.078432 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:01:01.078393    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:01:07.304680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.304644    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"]
Oct 30 22:01:07.305483 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.305466    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy"
Oct 30 22:01:07.305536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.305485    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb42d7c-2086-4352-a6b3-3e8f3c09da77" containerName="istio-proxy"
Oct 30 22:01:07.305536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.305532    6695 cpu_manager.go:414] "containerMap: removing container" podUID="15d36087-dcb8-4cb8-9c99-baebdc35f03a" containerName="revoke-keys"
Oct 30 22:01:07.305623 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.305538    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d36087-dcb8-4cb8-9c99-baebdc35f03a" containerName="revoke-keys"
Oct 30 22:01:07.305687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.305679    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d36087-dcb8-4cb8-9c99-baebdc35f03a" containerName="revoke-keys"
Oct 30 22:01:07.309795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.309773    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.333432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.333389    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"]
Oct 30 22:01:07.427718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.427679    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.427901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.427856    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.427959 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.427900    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.428059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.428037    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.529141 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.529101    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.529346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.529195    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.529346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.529222    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.529346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.529271    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.529942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.529913    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.530090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.530066    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.531881 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.531858    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.551296 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.551270    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk\") pod \"payload-processing-external-model-7d8545fb8d-stcz7\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") " pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:07.628009 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:07.627911    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:08.086944 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:08.086912    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"]
Oct 30 22:01:08.089379 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:01:08.089353    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091d14b6_6cb4_4a88_b570_627764ce3c70.slice/crio-dae3bc780ce6b8bf47857fd3cf5a2c2a042e9ef2430d82f49774e7f08edd33e8 WatchSource:0}: Error finding container dae3bc780ce6b8bf47857fd3cf5a2c2a042e9ef2430d82f49774e7f08edd33e8: Status 404 returned error can't find the container with id dae3bc780ce6b8bf47857fd3cf5a2c2a042e9ef2430d82f49774e7f08edd33e8
Oct 30 22:01:08.882365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:08.882285    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7" event={"ID":"091d14b6-6cb4-4a88-b570-627764ce3c70","Type":"ContainerStarted","Data":"d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde"}
Oct 30 22:01:08.882954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:08.882392    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7" event={"ID":"091d14b6-6cb4-4a88-b570-627764ce3c70","Type":"ContainerStarted","Data":"dae3bc780ce6b8bf47857fd3cf5a2c2a042e9ef2430d82f49774e7f08edd33e8"}
Oct 30 22:01:08.882954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:08.882428    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:08.940783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:08.940697    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7" podStartSLOduration=1.94061931 podStartE2EDuration="1.94061931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.936095177 +0000 UTC m=+3051.858553477" watchObservedRunningTime="2026-10-30 22:01:08.94061931 +0000 UTC m=+3051.863077691"
Oct 30 22:01:13.775745 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.775711    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"]
Oct 30 22:01:13.781325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.781303    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:13.797429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.797409    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"kube-root-ca.crt\""
Oct 30 22:01:13.809536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.809513    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"payload-processing-external-model-e2e--d5cdf106-dockercfg-mtgwn\""
Oct 30 22:01:13.809834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.809815    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"agc-cred-3018d9e8cbed\""
Oct 30 22:01:13.814029 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.814002    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"]
Oct 30 22:01:13.903408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.903361    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5d3601\""
Oct 30 22:01:13.903775 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.903753    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"openshift-service-ca.crt\""
Oct 30 22:01:13.904128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.904069    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"routing-overlay\""
Oct 30 22:01:13.908084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.906444    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:13.908084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.906493    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:13.908084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.906640    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:13.908084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:13.906694    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8bb\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.063090    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.063145    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.063296    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.063338    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8bb\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.064522    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.066922    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.069639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.067419    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.094447 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.094413    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8bb\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.099469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.097461    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:14.428738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.428702    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"]
Oct 30 22:01:14.431108 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:01:14.431077    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb6f597_caf3_427b_b9de_d8eb38a33b7d.slice/crio-618fd8b45d81e216a5648f7f0f623fe1c05f752c4e08a776f4c6947c1ad4346d WatchSource:0}: Error finding container 618fd8b45d81e216a5648f7f0f623fe1c05f752c4e08a776f4c6947c1ad4346d: Status 404 returned error can't find the container with id 618fd8b45d81e216a5648f7f0f623fe1c05f752c4e08a776f4c6947c1ad4346d
Oct 30 22:01:14.752487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.752408    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"]
Oct 30 22:01:14.757299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.757276    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.773231 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.773205    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"kube-root-ca.crt\""
Oct 30 22:01:14.773350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.773256    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"openshift-service-ca.crt\""
Oct 30 22:01:14.773350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.773282    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-ed981606a7c4\""
Oct 30 22:01:14.773350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.773299    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ed0866\""
Oct 30 22:01:14.773499 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.773346    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"routing-overlay\""
Oct 30 22:01:14.784752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.784727    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-e2e--68c3d057-dockercfg-2s4kt\""
Oct 30 22:01:14.790800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.790768    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"]
Oct 30 22:01:14.800807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.800777    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.800985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.800825    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.800985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.800902    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.801126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.801068    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsgb\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.925627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.921537    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.925627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.921804    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsgb\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.925627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.921935    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.925627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.921975    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.925627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.923982    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.938628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.934207    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.965626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.965123    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsgb\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:14.981623 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:14.968127    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:15.072650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:15.072231    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:15.089800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:15.089639    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4" event={"ID":"8fb6f597-caf3-427b-b9de-d8eb38a33b7d","Type":"ContainerStarted","Data":"a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669"}
Oct 30 22:01:15.089800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:15.089705    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4" event={"ID":"8fb6f597-caf3-427b-b9de-d8eb38a33b7d","Type":"ContainerStarted","Data":"618fd8b45d81e216a5648f7f0f623fe1c05f752c4e08a776f4c6947c1ad4346d"}
Oct 30 22:01:15.089800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:15.089766    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:15.135078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:15.134588    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4" podStartSLOduration=2.13457204 podStartE2EDuration="2.13457204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:15.12721909 +0000 UTC m=+3058.049677384" watchObservedRunningTime="2026-10-30 22:01:15.13457204 +0000 UTC m=+3058.057030339"
Oct 30 22:01:15.391100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:15.391075    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"]
Oct 30 22:01:15.393316 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:01:15.393290    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4069ebd_0efb_4cdd_84e9_dec11dda2dfc.slice/crio-f34b1f19a5435dfe0d99d4dca0c38557a90a1656d1183c2db469a6ffaa701edd WatchSource:0}: Error finding container f34b1f19a5435dfe0d99d4dca0c38557a90a1656d1183c2db469a6ffaa701edd: Status 404 returned error can't find the container with id f34b1f19a5435dfe0d99d4dca0c38557a90a1656d1183c2db469a6ffaa701edd
Oct 30 22:01:16.100443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:16.100407    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4" event={"ID":"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc","Type":"ContainerStarted","Data":"d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5"}
Oct 30 22:01:16.100962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:16.100464    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4" event={"ID":"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc","Type":"ContainerStarted","Data":"f34b1f19a5435dfe0d99d4dca0c38557a90a1656d1183c2db469a6ffaa701edd"}
Oct 30 22:01:16.100962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:16.100491    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:16.142397 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:16.142345    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4" podStartSLOduration=2.14233086 podStartE2EDuration="2.14233086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:16.141228187 +0000 UTC m=+3059.063686487" watchObservedRunningTime="2026-10-30 22:01:16.14233086 +0000 UTC m=+3059.064789158"
Oct 30 22:01:19.895875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:19.895844    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:01:19.969906 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:19.969865    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"]
Oct 30 22:01:23.323805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:23.323766    6695 scope.go:122] "RemoveContainer" containerID="b0343ae6b9e8f6b43e4840cf8d0e9a32d69244d271173256aec3858a1a4c4062"
Oct 30 22:01:23.334920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:23.334897    6695 scope.go:122] "RemoveContainer" containerID="a1e270e80d8d82c33a958f41bfe9ec95f059767d364b52977633a662c892b6ca"
Oct 30 22:01:23.345006 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:23.344981    6695 scope.go:122] "RemoveContainer" containerID="f01f465ac812ca0f280a8d399e2854f6166a785c720c1d34509e5704eb802da0"
Oct 30 22:01:23.355333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:23.355252    6695 scope.go:122] "RemoveContainer" containerID="99da5c6a31846b568b99638794a96df9c0d5b907222e96a34b69ece37259ae3c"
Oct 30 22:01:23.368841 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:23.368817    6695 scope.go:122] "RemoveContainer" containerID="09bdf8be777ee8db57b14b97beef2b519910143baca11ade9a3e5bc5299fa345"
Oct 30 22:01:24.572047 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.572013    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"]
Oct 30 22:01:24.578876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.578849    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.594778 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.594744    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"]
Oct 30 22:01:24.599424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.599400    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-4b2713\"/\"kube-root-ca.crt\""
Oct 30 22:01:24.604265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.604225    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-4b2713\"/\"openshift-service-ca.crt\""
Oct 30 22:01:24.604406 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.604367    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-4b2713\"/\"e2e-usage-sim-cea690-cert\""
Oct 30 22:01:24.604486 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.604449    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-4b2713\"/\"default-dockercfg-t8fzw\""
Oct 30 22:01:24.678387 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.678342    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls\") pod \"e2e-usage-sim-cea690-594dc6847c-ptbxh\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") " pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.678585 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.678507    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8bm\" (UniqueName: \"kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm\") pod \"e2e-usage-sim-cea690-594dc6847c-ptbxh\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") " pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.780404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.780362    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls\") pod \"e2e-usage-sim-cea690-594dc6847c-ptbxh\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") " pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.780634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.780493    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8bm\" (UniqueName: \"kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm\") pod \"e2e-usage-sim-cea690-594dc6847c-ptbxh\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") " pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.782832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.782806    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls\") pod \"e2e-usage-sim-cea690-594dc6847c-ptbxh\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") " pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.799682 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.799655    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8bm\" (UniqueName: \"kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm\") pod \"e2e-usage-sim-cea690-594dc6847c-ptbxh\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") " pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.894238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.894149    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:24.973409 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:24.973174    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l" podUID="f0a11982-bcde-40b4-9518-ccfab9564ae0" containerName="payload-processing" containerID="cri-o://e9133c6e3875aefaa88ae50ed32b426319788e8711c4bb336443ca8a42ef209a" gracePeriod=25
Oct 30 22:01:25.179061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.179027    6695 generic.go:356] "Generic (PLEG): container finished" podID="f0a11982-bcde-40b4-9518-ccfab9564ae0" containerID="e9133c6e3875aefaa88ae50ed32b426319788e8711c4bb336443ca8a42ef209a" exitCode=0
Oct 30 22:01:25.179238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.179100    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l" event={"ID":"f0a11982-bcde-40b4-9518-ccfab9564ae0","Type":"ContainerDied","Data":"e9133c6e3875aefaa88ae50ed32b426319788e8711c4bb336443ca8a42ef209a"}
Oct 30 22:01:25.216105 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:01:25.216069    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5ea70b_623d_4eda_b7cc_188cf3fe1fd6.slice/crio-8263c4bc350868ccea3dab4f7e3acee64a561376140a76673fbeeff5739fe129 WatchSource:0}: Error finding container 8263c4bc350868ccea3dab4f7e3acee64a561376140a76673fbeeff5739fe129: Status 404 returned error can't find the container with id 8263c4bc350868ccea3dab4f7e3acee64a561376140a76673fbeeff5739fe129
Oct 30 22:01:25.222618 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.222582    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"]
Oct 30 22:01:25.254353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.254329    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 22:01:25.286900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.286873    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn\") pod \"f0a11982-bcde-40b4-9518-ccfab9564ae0\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") "
Oct 30 22:01:25.287017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.286998    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume\") pod \"f0a11982-bcde-40b4-9518-ccfab9564ae0\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") "
Oct 30 22:01:25.287148 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.287132    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume\") pod \"f0a11982-bcde-40b4-9518-ccfab9564ae0\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") "
Oct 30 22:01:25.287323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.287299    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials\") pod \"f0a11982-bcde-40b4-9518-ccfab9564ae0\" (UID: \"f0a11982-bcde-40b4-9518-ccfab9564ae0\") "
Oct 30 22:01:25.287448 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.287401    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume" pod "f0a11982-bcde-40b4-9518-ccfab9564ae0" (UID: "f0a11982-bcde-40b4-9518-ccfab9564ae0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:25.287658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.287617    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume" pod "f0a11982-bcde-40b4-9518-ccfab9564ae0" (UID: "f0a11982-bcde-40b4-9518-ccfab9564ae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:25.288142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.288119    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:25.288256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.288149    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a11982-bcde-40b4-9518-ccfab9564ae0-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:25.289213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.289186    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn" pod "f0a11982-bcde-40b4-9518-ccfab9564ae0" (UID: "f0a11982-bcde-40b4-9518-ccfab9564ae0"). InnerVolumeSpecName "kube-api-access-mpptn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:25.289573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.289552    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials" pod "f0a11982-bcde-40b4-9518-ccfab9564ae0" (UID: "f0a11982-bcde-40b4-9518-ccfab9564ae0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:25.389484 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.389450    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:25.389484 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:25.389483    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpptn\" (UniqueName: \"kubernetes.io/projected/f0a11982-bcde-40b4-9518-ccfab9564ae0-kube-api-access-mpptn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:26.103965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.103882    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:26.192295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.192263    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"
Oct 30 22:01:26.192479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.192293    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f6d747c94-sf29l" event={"ID":"f0a11982-bcde-40b4-9518-ccfab9564ae0","Type":"ContainerDied","Data":"5949f4a9be504856afb01a135d9ccc5ec4c2df56a449a491d11e727fc54da505"}
Oct 30 22:01:26.192479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.192361    6695 scope.go:122] "RemoveContainer" containerID="e9133c6e3875aefaa88ae50ed32b426319788e8711c4bb336443ca8a42ef209a"
Oct 30 22:01:26.196240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.196210    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh" event={"ID":"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6","Type":"ContainerStarted","Data":"d90ce44e0f712ba9120313162685846cc04c6850fed55e272a4322ea3251dc54"}
Oct 30 22:01:26.196387 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.196260    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:26.196387 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.196279    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh" event={"ID":"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6","Type":"ContainerStarted","Data":"8263c4bc350868ccea3dab4f7e3acee64a561376140a76673fbeeff5739fe129"}
Oct 30 22:01:26.204169 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.203150    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:01:26.248569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.248534    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"]
Oct 30 22:01:26.255517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.255481    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f6d747c94-sf29l"]
Oct 30 22:01:26.297947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:26.297893    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh" podStartSLOduration=2.297878088 podStartE2EDuration="2.297878088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:26.295078988 +0000 UTC m=+3069.217537287" watchObservedRunningTime="2026-10-30 22:01:26.297878088 +0000 UTC m=+3069.220336415"
Oct 30 22:01:27.113463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:27.113431    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:27.217683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:27.217650    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a11982-bcde-40b4-9518-ccfab9564ae0" path="/var/lib/kubelet/pods/f0a11982-bcde-40b4-9518-ccfab9564ae0/volumes"
Oct 30 22:01:29.058682 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:29.058312    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"]
Oct 30 22:01:29.086627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:29.082300    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a67d34-2l8rs"]
Oct 30 22:01:29.319688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:29.319507    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2332cd-30fd-407f-9646-41786134fc7c" path="/var/lib/kubelet/pods/1a2332cd-30fd-407f-9646-41786134fc7c/volumes"
Oct 30 22:01:31.078597 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:01:31.078551    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:01:34.167426 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.167391    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"]
Oct 30 22:01:34.168264 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.168246    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f0a11982-bcde-40b4-9518-ccfab9564ae0" containerName="payload-processing"
Oct 30 22:01:34.168320 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.168265    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a11982-bcde-40b4-9518-ccfab9564ae0" containerName="payload-processing"
Oct 30 22:01:34.168408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.168399    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a11982-bcde-40b4-9518-ccfab9564ae0" containerName="payload-processing"
Oct 30 22:01:34.173873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.173854    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.191381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.191296    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"]
Oct 30 22:01:34.197287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.197258    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"agc-cred-b4e7f195a991\""
Oct 30 22:01:34.276528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.276485    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnshd\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.276774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.276575    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.276861 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.276829    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.276904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.276879    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.378400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.378368    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.378633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.378424    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.378633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.378570    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnshd\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.378794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.378635    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.379266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.379235    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.379360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.379245    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.381089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.381065    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.397520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.397494    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnshd\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.490273 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.490200    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:34.796187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:34.796135    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"]
Oct 30 22:01:34.799262 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:01:34.799229    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c336a4c_d2a7_459f_b1e9_a2f79109c186.slice/crio-f9be71f02b16e987b9562c40952335efc607c61ef158e9370eb12bcd5bdb9390 WatchSource:0}: Error finding container f9be71f02b16e987b9562c40952335efc607c61ef158e9370eb12bcd5bdb9390: Status 404 returned error can't find the container with id f9be71f02b16e987b9562c40952335efc607c61ef158e9370eb12bcd5bdb9390
Oct 30 22:01:35.351479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:35.351440    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx" event={"ID":"9c336a4c-d2a7-459f-b1e9-a2f79109c186","Type":"ContainerStarted","Data":"6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34"}
Oct 30 22:01:35.351917 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:35.351512    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:35.351917 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:35.351531    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx" event={"ID":"9c336a4c-d2a7-459f-b1e9-a2f79109c186","Type":"ContainerStarted","Data":"f9be71f02b16e987b9562c40952335efc607c61ef158e9370eb12bcd5bdb9390"}
Oct 30 22:01:35.392260 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:35.392193    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx" podStartSLOduration=1.392175573 podStartE2EDuration="1.392175573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:35.388370719 +0000 UTC m=+3078.310829016" watchObservedRunningTime="2026-10-30 22:01:35.392175573 +0000 UTC m=+3078.314633871"
Oct 30 22:01:42.034660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:42.034616    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"]
Oct 30 22:01:42.045931 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:42.045903    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a679259f-gxqjb"]
Oct 30 22:01:43.222548 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:43.222509    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7012197-6304-44e7-881b-ba0a661a5aa0" path="/var/lib/kubelet/pods/f7012197-6304-44e7-881b-ba0a661a5aa0/volumes"
Oct 30 22:01:45.820699 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:45.820664    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"]
Oct 30 22:01:46.363095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:46.363069    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:01:46.439506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:46.439465    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"]
Oct 30 22:01:50.821662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:50.821619    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4" podUID="f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" containerName="payload-processing" containerID="cri-o://d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5" gracePeriod=25
Oct 30 22:01:51.104578 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.104549    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:51.161282 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.161241    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials\") pod \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") "
Oct 30 22:01:51.161503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.161469    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb\") pod \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") "
Oct 30 22:01:51.161674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.161521    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume\") pod \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") "
Oct 30 22:01:51.161674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.161660    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume\") pod \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\" (UID: \"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc\") "
Oct 30 22:01:51.162945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.162833    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume" pod "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" (UID: "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:51.162945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.162834    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume" pod "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" (UID: "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:51.164673 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.164648    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb" pod "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" (UID: "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc"). InnerVolumeSpecName "kube-api-access-rrsgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:51.164803 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.164685    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials" pod "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" (UID: "f4069ebd-0efb-4cdd-84e9-dec11dda2dfc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:51.264463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.264431    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrsgb\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-kube-api-access-rrsgb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.264463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.264462    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.264737 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.264475    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.264737 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.264488    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.440565 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.440461    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4" podUID="8fb6f597-caf3-427b-b9de-d8eb38a33b7d" containerName="payload-processing" containerID="cri-o://a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669" gracePeriod=25
Oct 30 22:01:51.488628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.488562    6695 generic.go:356] "Generic (PLEG): container finished" podID="f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" containerID="d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5" exitCode=0
Oct 30 22:01:51.488784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.488668    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"
Oct 30 22:01:51.488784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.488664    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4" event={"ID":"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc","Type":"ContainerDied","Data":"d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5"}
Oct 30 22:01:51.488904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.488798    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4" event={"ID":"f4069ebd-0efb-4cdd-84e9-dec11dda2dfc","Type":"ContainerDied","Data":"f34b1f19a5435dfe0d99d4dca0c38557a90a1656d1183c2db469a6ffaa701edd"}
Oct 30 22:01:51.488904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.488822    6695 scope.go:122] "RemoveContainer" containerID="d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5"
Oct 30 22:01:51.549733 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.549707    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"]
Oct 30 22:01:51.554370 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.554336    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5q68f4"]
Oct 30 22:01:51.555322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.555301    6695 scope.go:122] "RemoveContainer" containerID="d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5"
Oct 30 22:01:51.555689 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:01:51.555669    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5\": container with ID starting with d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5 not found: ID does not exist" containerID="d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5"
Oct 30 22:01:51.555771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.555698    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5"} err="failed to get container status \"d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5\": rpc error: code = NotFound desc = could not find container \"d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5\": container with ID starting with d0afdc42c999bc51e2342fdb2cf6fc2fc5fa5b69c28b0114b0628da6bd55feb5 not found: ID does not exist"
Oct 30 22:01:51.692984 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.692924    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:51.882576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.882542    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume\") pod \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") "
Oct 30 22:01:51.883055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.882636    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume\") pod \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") "
Oct 30 22:01:51.883055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.882743    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb\") pod \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") "
Oct 30 22:01:51.883055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.882866    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials\") pod \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\" (UID: \"8fb6f597-caf3-427b-b9de-d8eb38a33b7d\") "
Oct 30 22:01:51.883218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.883057    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume" pod "8fb6f597-caf3-427b-b9de-d8eb38a33b7d" (UID: "8fb6f597-caf3-427b-b9de-d8eb38a33b7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:51.883218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.883105    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume" pod "8fb6f597-caf3-427b-b9de-d8eb38a33b7d" (UID: "8fb6f597-caf3-427b-b9de-d8eb38a33b7d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:51.883534 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.883517    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.883587 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.883537    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.885117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.885095    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials" pod "8fb6f597-caf3-427b-b9de-d8eb38a33b7d" (UID: "8fb6f597-caf3-427b-b9de-d8eb38a33b7d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:51.885206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.885169    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb" pod "8fb6f597-caf3-427b-b9de-d8eb38a33b7d" (UID: "8fb6f597-caf3-427b-b9de-d8eb38a33b7d"). InnerVolumeSpecName "kube-api-access-6p8bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:51.984450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.984367    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p8bb\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-kube-api-access-6p8bb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:51.984450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:51.984408    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fb6f597-caf3-427b-b9de-d8eb38a33b7d-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:01:52.499377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.499342    6695 generic.go:356] "Generic (PLEG): container finished" podID="8fb6f597-caf3-427b-b9de-d8eb38a33b7d" containerID="a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669" exitCode=0
Oct 30 22:01:52.499582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.499391    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"
Oct 30 22:01:52.499582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.499428    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4" event={"ID":"8fb6f597-caf3-427b-b9de-d8eb38a33b7d","Type":"ContainerDied","Data":"a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669"}
Oct 30 22:01:52.499582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.499480    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4" event={"ID":"8fb6f597-caf3-427b-b9de-d8eb38a33b7d","Type":"ContainerDied","Data":"618fd8b45d81e216a5648f7f0f623fe1c05f752c4e08a776f4c6947c1ad4346d"}
Oct 30 22:01:52.499582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.499504    6695 scope.go:122] "RemoveContainer" containerID="a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669"
Oct 30 22:01:52.511316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.511298    6695 scope.go:122] "RemoveContainer" containerID="a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669"
Oct 30 22:01:52.511619 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:01:52.511589    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669\": container with ID starting with a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669 not found: ID does not exist" containerID="a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669"
Oct 30 22:01:52.511687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.511632    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669"} err="failed to get container status \"a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669\": rpc error: code = NotFound desc = could not find container \"a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669\": container with ID starting with a42592990c280b2f0d1381c651556728701bd63a6bafcd59fd452039fb826669 not found: ID does not exist"
Oct 30 22:01:52.551112 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.551081    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"]
Oct 30 22:01:52.562325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:52.562301    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5d2vn4"]
Oct 30 22:01:53.217318 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:53.217283    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb6f597-caf3-427b-b9de-d8eb38a33b7d" path="/var/lib/kubelet/pods/8fb6f597-caf3-427b-b9de-d8eb38a33b7d/volumes"
Oct 30 22:01:53.217801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:53.217785    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" path="/var/lib/kubelet/pods/f4069ebd-0efb-4cdd-84e9-dec11dda2dfc/volumes"
Oct 30 22:01:54.035051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:54.035014    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"]
Oct 30 22:01:54.045353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:54.045324    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0632f1f3-x7mrg"]
Oct 30 22:01:55.215509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:01:55.215478    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99eb095b-0823-4fd4-8c18-8cc1680ffc71" path="/var/lib/kubelet/pods/99eb095b-0823-4fd4-8c18-8cc1680ffc71/volumes"
Oct 30 22:02:01.078655 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:02:01.078617    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:02:08.028167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:08.028131    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"]
Oct 30 22:02:08.039114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:08.039074    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb403179-s25r6"]
Oct 30 22:02:09.217881 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:09.217845    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1f3a02-4f77-4290-b3c3-f1f6e32e308c" path="/var/lib/kubelet/pods/7c1f3a02-4f77-4290-b3c3-f1f6e32e308c/volumes"
Oct 30 22:02:18.064888 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:18.064843    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"]
Oct 30 22:02:18.075964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:18.075925    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d76819-84h5p"]
Oct 30 22:02:19.217127 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:19.217084    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208a18cc-9f25-4f0f-bf0a-e39f55bd29d9" path="/var/lib/kubelet/pods/208a18cc-9f25-4f0f-bf0a-e39f55bd29d9/volumes"
Oct 30 22:02:23.442952 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:23.442910    6695 scope.go:122] "RemoveContainer" containerID="1ea665ea2bdc0ad135a02f927f12122b5507eb87a326bdbe5d6b60fccf120690"
Oct 30 22:02:31.076222 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:02:31.076181    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:02:38.028384 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:38.028331    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"]
Oct 30 22:02:38.039586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:38.039549    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f869ca42-6bqm7"]
Oct 30 22:02:39.216840 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:39.216799    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1eb3d60-953b-4af5-9420-8870674004b2" path="/var/lib/kubelet/pods/c1eb3d60-953b-4af5-9420-8870674004b2/volumes"
Oct 30 22:02:43.027642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:43.027581    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"]
Oct 30 22:02:43.038310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:43.038269    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f2d897-2p27f"]
Oct 30 22:02:43.219043 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:43.218999    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3000f88e-b5f3-4b52-a2b4-d75063792c4b" path="/var/lib/kubelet/pods/3000f88e-b5f3-4b52-a2b4-d75063792c4b/volumes"
Oct 30 22:02:49.031138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:49.031098    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"]
Oct 30 22:02:49.041674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:49.041596    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4799f-8sbcr"]
Oct 30 22:02:49.217299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:02:49.217253    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b17a2c-eb3e-4d0b-bbac-7e845ff4960f" path="/var/lib/kubelet/pods/79b17a2c-eb3e-4d0b-bbac-7e845ff4960f/volumes"
Oct 30 22:03:01.079019 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:01.078914    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:03:07.511503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.511460    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"]
Oct 30 22:03:07.512353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.512333    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8fb6f597-caf3-427b-b9de-d8eb38a33b7d" containerName="payload-processing"
Oct 30 22:03:07.512401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.512354    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb6f597-caf3-427b-b9de-d8eb38a33b7d" containerName="payload-processing"
Oct 30 22:03:07.512401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.512390    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" containerName="payload-processing"
Oct 30 22:03:07.512401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.512395    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" containerName="payload-processing"
Oct 30 22:03:07.512511 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.512503    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb6f597-caf3-427b-b9de-d8eb38a33b7d" containerName="payload-processing"
Oct 30 22:03:07.512551 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.512522    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4069ebd-0efb-4cdd-84e9-dec11dda2dfc" containerName="payload-processing"
Oct 30 22:03:07.516810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.516777    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.545998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.545954    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:07.550042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.549938    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:03:07.550429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.550067    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:07.550429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.550140    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:07.556078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.556033    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"]
Oct 30 22:03:07.623271 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.623230    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pk6\" (UniqueName: \"kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.623477 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.623336    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.623533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.623460    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.725445 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.725398    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.725686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.725630    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pk6\" (UniqueName: \"kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.725759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.725699    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.726339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.726304    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.728472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.728441    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.744897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.744855    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pk6\" (UniqueName: \"kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6\") pod \"maas-api-revoke-sub-9507eef9-95699\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:07.834424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:07.834383    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:08.202925 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:08.202706    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"]
Oct 30 22:03:08.206655 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:08.206585    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf589d7_66a3_495d_8e89_05ac79973334.slice/crio-3bd37aad594bda55f8f1170d2f7bafc3dd799474cdd3332fdad14f92236c3c37 WatchSource:0}: Error finding container 3bd37aad594bda55f8f1170d2f7bafc3dd799474cdd3332fdad14f92236c3c37: Status 404 returned error can't find the container with id 3bd37aad594bda55f8f1170d2f7bafc3dd799474cdd3332fdad14f92236c3c37
Oct 30 22:03:08.228049 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:08.227865    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699" event={"ID":"fdf589d7-66a3-495d-8e89-05ac79973334","Type":"ContainerStarted","Data":"3bd37aad594bda55f8f1170d2f7bafc3dd799474cdd3332fdad14f92236c3c37"}
Oct 30 22:03:09.238427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:09.238386    6695 generic.go:356] "Generic (PLEG): container finished" podID="fdf589d7-66a3-495d-8e89-05ac79973334" containerID="8ec9011ab0a53d6c9468210265114fd4d1be8ec3c4a1d1bfb0cea7969773d0da" exitCode=0
Oct 30 22:03:09.238427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:09.238429    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699" event={"ID":"fdf589d7-66a3-495d-8e89-05ac79973334","Type":"ContainerDied","Data":"8ec9011ab0a53d6c9468210265114fd4d1be8ec3c4a1d1bfb0cea7969773d0da"}
Oct 30 22:03:10.450153 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.450110    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:10.574465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.574419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6\" (UniqueName: \"kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6\") pod \"fdf589d7-66a3-495d-8e89-05ac79973334\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") "
Oct 30 22:03:10.574759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.574667    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca\") pod \"fdf589d7-66a3-495d-8e89-05ac79973334\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") "
Oct 30 22:03:10.574759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.574699    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca\") pod \"fdf589d7-66a3-495d-8e89-05ac79973334\" (UID: \"fdf589d7-66a3-495d-8e89-05ac79973334\") "
Oct 30 22:03:10.575085 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.575045    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca" pod "fdf589d7-66a3-495d-8e89-05ac79973334" (UID: "fdf589d7-66a3-495d-8e89-05ac79973334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:10.575439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.575418    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:10.576977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.576947    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca" pod "fdf589d7-66a3-495d-8e89-05ac79973334" (UID: "fdf589d7-66a3-495d-8e89-05ac79973334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:10.577100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.576995    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6" pod "fdf589d7-66a3-495d-8e89-05ac79973334" (UID: "fdf589d7-66a3-495d-8e89-05ac79973334"). InnerVolumeSpecName "kube-api-access-55pk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:10.677645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.677585    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf589d7-66a3-495d-8e89-05ac79973334-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:10.677645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:10.677642    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pk6\" (UniqueName: \"kubernetes.io/projected/fdf589d7-66a3-495d-8e89-05ac79973334-kube-api-access-55pk6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:11.256791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:11.256752    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699" event={"ID":"fdf589d7-66a3-495d-8e89-05ac79973334","Type":"ContainerDied","Data":"3bd37aad594bda55f8f1170d2f7bafc3dd799474cdd3332fdad14f92236c3c37"}
Oct 30 22:03:11.256791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:11.256787    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"
Oct 30 22:03:11.257092 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:11.256808    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bd37aad594bda55f8f1170d2f7bafc3dd799474cdd3332fdad14f92236c3c37"
Oct 30 22:03:13.973442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:13.973395    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"]
Oct 30 22:03:13.974748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:13.974724    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fdf589d7-66a3-495d-8e89-05ac79973334" containerName="revoke-keys"
Oct 30 22:03:13.974748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:13.974749    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf589d7-66a3-495d-8e89-05ac79973334" containerName="revoke-keys"
Oct 30 22:03:13.974910 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:13.974900    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf589d7-66a3-495d-8e89-05ac79973334" containerName="revoke-keys"
Oct 30 22:03:13.979038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:13.979003    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:13.998681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:13.996532    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"]
Oct 30 22:03:14.005628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.002654    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:14.005628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.002664    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:14.005628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.002719    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:14.005927 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.005743    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:03:14.031802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.031752    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.032005 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.031817    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bzg\" (UniqueName: \"kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.032005 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.031912    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.133490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.133437    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.133490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.133492    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bzg\" (UniqueName: \"kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.133863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.133552    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.134408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.134377    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.136206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.136173    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.152908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.152872    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bzg\" (UniqueName: \"kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg\") pod \"maas-api-revoke-sub-253a5db0-rrvh6\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.299112 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.299065    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:14.617252 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:14.617211    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"]
Oct 30 22:03:14.619772 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:14.619737    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a2bdd5_7363_4526_b7a3_29f5b26c41d9.slice/crio-e03ee9aefa1073bbc55eaf8978841f8fb8c23ed6f787dd724ea82c7dc54c239e WatchSource:0}: Error finding container e03ee9aefa1073bbc55eaf8978841f8fb8c23ed6f787dd724ea82c7dc54c239e: Status 404 returned error can't find the container with id e03ee9aefa1073bbc55eaf8978841f8fb8c23ed6f787dd724ea82c7dc54c239e
Oct 30 22:03:15.294354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:15.294250    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6" event={"ID":"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9","Type":"ContainerStarted","Data":"1bff3188029bae406f78316d7c14e339c30843508e741a444b4d0374514c68e1"}
Oct 30 22:03:15.294354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:15.294322    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6" event={"ID":"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9","Type":"ContainerStarted","Data":"e03ee9aefa1073bbc55eaf8978841f8fb8c23ed6f787dd724ea82c7dc54c239e"}
Oct 30 22:03:15.334753 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:15.334684    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6" podStartSLOduration=2.334663498 podStartE2EDuration="2.334663498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:15.33130731 +0000 UTC m=+3178.253765608" watchObservedRunningTime="2026-10-30 22:03:15.334663498 +0000 UTC m=+3178.257121795"
Oct 30 22:03:16.304526 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:16.304486    6695 generic.go:356] "Generic (PLEG): container finished" podID="d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" containerID="1bff3188029bae406f78316d7c14e339c30843508e741a444b4d0374514c68e1" exitCode=0
Oct 30 22:03:16.305010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:16.304576    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6" event={"ID":"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9","Type":"ContainerDied","Data":"1bff3188029bae406f78316d7c14e339c30843508e741a444b4d0374514c68e1"}
Oct 30 22:03:17.153175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.153126    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"]
Oct 30 22:03:17.153396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.153377    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy" containerID="cri-o://43c96c7324a062394d32df659325f98e51329fdc5b446bb86aef29bf410d9989" gracePeriod=30
Oct 30 22:03:17.513872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.513830    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:17.595854 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.595812    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca\") pod \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") "
Oct 30 22:03:17.596041 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.595923    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca\") pod \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") "
Oct 30 22:03:17.596244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.596222    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg\" (UniqueName: \"kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg\") pod \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\" (UID: \"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9\") "
Oct 30 22:03:17.596348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.596240    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca" pod "d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" (UID: "d9a2bdd5-7363-4526-b7a3-29f5b26c41d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:17.596955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.596928    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:17.604669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.598997    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg" pod "d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" (UID: "d9a2bdd5-7363-4526-b7a3-29f5b26c41d9"). InnerVolumeSpecName "kube-api-access-c2bzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:17.604669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.599201    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca" pod "d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" (UID: "d9a2bdd5-7363-4526-b7a3-29f5b26c41d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:17.698451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.698399    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2bzg\" (UniqueName: \"kubernetes.io/projected/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-kube-api-access-c2bzg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:17.698451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:17.698443    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:18.219537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.219488    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"]
Oct 30 22:03:18.220405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.220385    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" containerName="revoke-keys"
Oct 30 22:03:18.220485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.220406    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" containerName="revoke-keys"
Oct 30 22:03:18.220583 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.220572    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" containerName="revoke-keys"
Oct 30 22:03:18.224917 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.224887    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.245376 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.245333    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"]
Oct 30 22:03:18.308986 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.308943    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27fz\" (UniqueName: \"kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.309190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.309116    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.309292 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.309262    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.326213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.326176    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6" event={"ID":"d9a2bdd5-7363-4526-b7a3-29f5b26c41d9","Type":"ContainerDied","Data":"e03ee9aefa1073bbc55eaf8978841f8fb8c23ed6f787dd724ea82c7dc54c239e"}
Oct 30 22:03:18.326213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.326210    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"
Oct 30 22:03:18.326413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.326228    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03ee9aefa1073bbc55eaf8978841f8fb8c23ed6f787dd724ea82c7dc54c239e"
Oct 30 22:03:18.411392 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.411334    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27fz\" (UniqueName: \"kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.411648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.411528    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.411754 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.411728    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.412293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.412266    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.414284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.414254    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.437680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.437638    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27fz\" (UniqueName: \"kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz\") pod \"maas-api-revoke-sub-3053b38e-v5dfw\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.549446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.549399    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:18.874876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:18.874830    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"]
Oct 30 22:03:18.877254 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:18.877219    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093f5235_281a_4726_8444_17047ac4d948.slice/crio-e980088b4505230fbad3d87512ce7d5e7d32809b6450ae58152d67ae03293d04 WatchSource:0}: Error finding container e980088b4505230fbad3d87512ce7d5e7d32809b6450ae58152d67ae03293d04: Status 404 returned error can't find the container with id e980088b4505230fbad3d87512ce7d5e7d32809b6450ae58152d67ae03293d04
Oct 30 22:03:19.337175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:19.337131    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw" event={"ID":"093f5235-281a-4726-8444-17047ac4d948","Type":"ContainerStarted","Data":"316beeea8e6e98cbc39fedcf5ae84c7b25b637d54a96a2733ad0b230866d704e"}
Oct 30 22:03:19.337355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:19.337203    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw" event={"ID":"093f5235-281a-4726-8444-17047ac4d948","Type":"ContainerStarted","Data":"e980088b4505230fbad3d87512ce7d5e7d32809b6450ae58152d67ae03293d04"}
Oct 30 22:03:19.375386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:19.375320    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw" podStartSLOduration=1.375303273 podStartE2EDuration="1.375303273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:19.373882925 +0000 UTC m=+3182.296341222" watchObservedRunningTime="2026-10-30 22:03:19.375303273 +0000 UTC m=+3182.297761571"
Oct 30 22:03:20.348870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:20.348816    6695 generic.go:356] "Generic (PLEG): container finished" podID="093f5235-281a-4726-8444-17047ac4d948" containerID="316beeea8e6e98cbc39fedcf5ae84c7b25b637d54a96a2733ad0b230866d704e" exitCode=0
Oct 30 22:03:20.349349 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:20.348897    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw" event={"ID":"093f5235-281a-4726-8444-17047ac4d948","Type":"ContainerDied","Data":"316beeea8e6e98cbc39fedcf5ae84c7b25b637d54a96a2733ad0b230866d704e"}
Oct 30 22:03:21.553867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.553831    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:21.672317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.672205    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz\" (UniqueName: \"kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz\") pod \"093f5235-281a-4726-8444-17047ac4d948\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") "
Oct 30 22:03:21.672317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.672303    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca\") pod \"093f5235-281a-4726-8444-17047ac4d948\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") "
Oct 30 22:03:21.672591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.672435    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca\") pod \"093f5235-281a-4726-8444-17047ac4d948\" (UID: \"093f5235-281a-4726-8444-17047ac4d948\") "
Oct 30 22:03:21.673407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.673368    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca" pod "093f5235-281a-4726-8444-17047ac4d948" (UID: "093f5235-281a-4726-8444-17047ac4d948"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:21.677758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.677573    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz" pod "093f5235-281a-4726-8444-17047ac4d948" (UID: "093f5235-281a-4726-8444-17047ac4d948"). InnerVolumeSpecName "kube-api-access-j27fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:21.680342 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.680290    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca" pod "093f5235-281a-4726-8444-17047ac4d948" (UID: "093f5235-281a-4726-8444-17047ac4d948"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:21.774643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.774579    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27fz\" (UniqueName: \"kubernetes.io/projected/093f5235-281a-4726-8444-17047ac4d948-kube-api-access-j27fz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:21.774643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.774630    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093f5235-281a-4726-8444-17047ac4d948-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:21.774643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:21.774642    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093f5235-281a-4726-8444-17047ac4d948-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.373696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.373269    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"
Oct 30 22:03:22.373696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.373276    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw" event={"ID":"093f5235-281a-4726-8444-17047ac4d948","Type":"ContainerDied","Data":"e980088b4505230fbad3d87512ce7d5e7d32809b6450ae58152d67ae03293d04"}
Oct 30 22:03:22.373696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.373410    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e980088b4505230fbad3d87512ce7d5e7d32809b6450ae58152d67ae03293d04"
Oct 30 22:03:22.376080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.376049    6695 generic.go:356] "Generic (PLEG): container finished" podID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerID="43c96c7324a062394d32df659325f98e51329fdc5b446bb86aef29bf410d9989" exitCode=0
Oct 30 22:03:22.376265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.376105    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" event={"ID":"bbb51624-01d8-48ed-9911-99ea9d0d8207","Type":"ContainerDied","Data":"43c96c7324a062394d32df659325f98e51329fdc5b446bb86aef29bf410d9989"}
Oct 30 22:03:22.376265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.376159    6695 scope.go:122] "RemoveContainer" containerID="b19fbf4b4a54a4cee257a703cca32fd3ef790f6d5270ad5e1799c9219dfda6c0"
Oct 30 22:03:22.431746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.431661    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.587201    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.589428    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.589572    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.589694    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.589749    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.590005    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.590089    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.590199    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.593908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.590308    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token\") pod \"bbb51624-01d8-48ed-9911-99ea9d0d8207\" (UID: \"bbb51624-01d8-48ed-9911-99ea9d0d8207\") "
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.589324    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.591626    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.591827    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.592232    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.596737    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.597465    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:22.597820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.597550    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:22.598220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.598146    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "kube-api-access-5hc22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:22.599410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.599381    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy" pod "bbb51624-01d8-48ed-9911-99ea9d0d8207" (UID: "bbb51624-01d8-48ed-9911-99ea9d0d8207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695765    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695801    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695814    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695824    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb51624-01d8-48ed-9911-99ea9d0d8207-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695833    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695840    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695849    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.695855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695858    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb51624-01d8-48ed-9911-99ea9d0d8207-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:22.696290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:22.695866    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hc22\" (UniqueName: \"kubernetes.io/projected/bbb51624-01d8-48ed-9911-99ea9d0d8207-kube-api-access-5hc22\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:23.387012 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.386976    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"
Oct 30 22:03:23.387209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.386977    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn" event={"ID":"bbb51624-01d8-48ed-9911-99ea9d0d8207","Type":"ContainerDied","Data":"b2a24ce8bab90b8106f81945c19d89b382f8e5aa645b93d0084771bbf069ad85"}
Oct 30 22:03:23.387209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.387121    6695 scope.go:122] "RemoveContainer" containerID="43c96c7324a062394d32df659325f98e51329fdc5b446bb86aef29bf410d9989"
Oct 30 22:03:23.451090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.451045    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"]
Oct 30 22:03:23.462301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.462257    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-cvpwn"]
Oct 30 22:03:23.496661 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.496556    6695 scope.go:122] "RemoveContainer" containerID="10fbea976dc0a88c456eaefd1194e51de3630a4962e214453af2f94dd0ad1901"
Oct 30 22:03:23.508943 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.508918    6695 scope.go:122] "RemoveContainer" containerID="b97038c9901ead34e195994fbc70ec70bd501ee5ba6e3439d125e12f9133eef4"
Oct 30 22:03:23.521205 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.521178    6695 scope.go:122] "RemoveContainer" containerID="8b1c1b4d00aa79404bf047e8a54dff023d4fc8dbc115666481465390fe89d643"
Oct 30 22:03:23.534140 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.534114    6695 scope.go:122] "RemoveContainer" containerID="a311063393769708c7fdf88ddbc392a678612692081e5f9090e60a77771ec8d6"
Oct 30 22:03:23.546001 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:23.545974    6695 scope.go:122] "RemoveContainer" containerID="dbce35fc3892e5eab973b0bdd144b8ab73ac6a42ebed57c0ccc27f2f97123b91"
Oct 30 22:03:25.217523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.217480    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" path="/var/lib/kubelet/pods/bbb51624-01d8-48ed-9911-99ea9d0d8207/volumes"
Oct 30 22:03:25.633466 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.633421    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"]
Oct 30 22:03:25.634514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634483    6695 cpu_manager.go:414] "containerMap: removing container" podUID="093f5235-281a-4726-8444-17047ac4d948" containerName="revoke-keys"
Oct 30 22:03:25.634514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634511    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093f5235-281a-4726-8444-17047ac4d948" containerName="revoke-keys"
Oct 30 22:03:25.634716 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634550    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy"
Oct 30 22:03:25.634716 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634557    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy"
Oct 30 22:03:25.634716 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634579    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy"
Oct 30 22:03:25.634716 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634585    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy"
Oct 30 22:03:25.634856 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634741    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="093f5235-281a-4726-8444-17047ac4d948" containerName="revoke-keys"
Oct 30 22:03:25.634856 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.634757    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy"
Oct 30 22:03:25.641078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.641045    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.657886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.657780    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"]
Oct 30 22:03:25.750360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.750310    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.750574 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.750373    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.750574 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.750466    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.750781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.750760    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.852560 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.852512    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.852799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.852649    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.852799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.852709    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.852799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.852751    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.853432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.853402    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.853599 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.853567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.856062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.856027    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.872562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.872520    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9\") pod \"payload-processing-external-model-6cb6ff46b5-6xndf\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:25.959304 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:25.959196    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:26.465513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:26.465465    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"]
Oct 30 22:03:26.469029 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:26.468994    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0c1245_a8a4_464e_a9a4_e808d2919250.slice/crio-4761be30b2ddb2daf81fb1bdb72b904cc8591f2bd13e6c85496b983a6464d086 WatchSource:0}: Error finding container 4761be30b2ddb2daf81fb1bdb72b904cc8591f2bd13e6c85496b983a6464d086: Status 404 returned error can't find the container with id 4761be30b2ddb2daf81fb1bdb72b904cc8591f2bd13e6c85496b983a6464d086
Oct 30 22:03:26.867875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:26.867817    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"]
Oct 30 22:03:26.868119 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:26.868049    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy" containerID="cri-o://6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e" gracePeriod=30
Oct 30 22:03:27.472818 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:27.472775    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf" event={"ID":"ce0c1245-a8a4-464e-a9a4-e808d2919250","Type":"ContainerStarted","Data":"a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2"}
Oct 30 22:03:27.473310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:27.472851    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf" event={"ID":"ce0c1245-a8a4-464e-a9a4-e808d2919250","Type":"ContainerStarted","Data":"4761be30b2ddb2daf81fb1bdb72b904cc8591f2bd13e6c85496b983a6464d086"}
Oct 30 22:03:27.473310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:27.472878    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:27.543391 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:27.543335    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf" podStartSLOduration=2.543319333 podStartE2EDuration="2.543319333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:27.542395174 +0000 UTC m=+3190.464853472" watchObservedRunningTime="2026-10-30 22:03:27.543319333 +0000 UTC m=+3190.465777631"
Oct 30 22:03:28.591419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.591374    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"]
Oct 30 22:03:28.625102 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.625057    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"]
Oct 30 22:03:28.626729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.626690    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb51624-01d8-48ed-9911-99ea9d0d8207" containerName="istio-proxy"
Oct 30 22:03:28.632053 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.632016    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.647555 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.647509    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"]
Oct 30 22:03:28.653538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.653502    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-73353bd0e503\""
Oct 30 22:03:28.676892 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.676824    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4chj\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.677261 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.677229    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.677380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.677312    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.677498 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.677479    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.782443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.782394    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.782443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.782454    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.782804 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.782480    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.782804 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.782591    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4chj\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.783470 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.783414    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.783684 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.783439    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.785961 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.785932    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.801896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.801862    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4chj\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj\") pod \"payload-processing-external-model-5ff676d685-2w8s7\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:28.952257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:28.952156    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:29.319826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:29.319796    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"]
Oct 30 22:03:29.323526 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:29.323486    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885099a0_6a17_4214_b132_a57b5c1d225a.slice/crio-ffdf2b85973cf0109de7dc15decbe9dcb27aecc7a5b7a27ee570d6e214bc1dc2 WatchSource:0}: Error finding container ffdf2b85973cf0109de7dc15decbe9dcb27aecc7a5b7a27ee570d6e214bc1dc2: Status 404 returned error can't find the container with id ffdf2b85973cf0109de7dc15decbe9dcb27aecc7a5b7a27ee570d6e214bc1dc2
Oct 30 22:03:29.496157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:29.496114    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7" event={"ID":"885099a0-6a17-4214-b132-a57b5c1d225a","Type":"ContainerStarted","Data":"1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3"}
Oct 30 22:03:29.496372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:29.496191    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7" event={"ID":"885099a0-6a17-4214-b132-a57b5c1d225a","Type":"ContainerStarted","Data":"ffdf2b85973cf0109de7dc15decbe9dcb27aecc7a5b7a27ee570d6e214bc1dc2"}
Oct 30 22:03:29.496372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:29.496222    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:29.536423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:29.536314    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7" podStartSLOduration=1.5362912990000002 podStartE2EDuration="1.536291299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:29.533788058 +0000 UTC m=+3192.456246355" watchObservedRunningTime="2026-10-30 22:03:29.536291299 +0000 UTC m=+3192.458749598"
Oct 30 22:03:31.231414 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:31.231367    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:03:31.248785 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.248744    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"]
Oct 30 22:03:31.254042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.254006    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.270686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.270652    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 22:03:31.282044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.281992    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 22:03:31.282044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.282001    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-010bb35e298d\""
Oct 30 22:03:31.282044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.282039    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 22:03:31.282344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.282125    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-pxsz6\""
Oct 30 22:03:31.286731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.286693    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"]
Oct 30 22:03:31.339067 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.339008    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsgtw\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.339310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.339098    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.339310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.339226    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.339430 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.339372    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.343660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.343630    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 22:03:31.446628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.441641    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.446628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.442024    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgtw\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.446628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.442137    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.446628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.442175    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.446628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.443301    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.446628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.445210    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.473635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.455483    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.475960 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.475902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgtw\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.577817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.577780    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:31.947466 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:31.947434    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"]
Oct 30 22:03:32.267313 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.267035    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"]
Oct 30 22:03:32.275887 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.275840    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"]
Oct 30 22:03:32.276133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.276016    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.291688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.291594    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"routing-overlay\""
Oct 30 22:03:32.291852 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.291708    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-f060d103b140\""
Oct 30 22:03:32.291852 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.291727    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"openshift-service-ca.crt\""
Oct 30 22:03:32.291852 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.291783    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-e2e--f95d90a5-dockercfg-5z6mv\""
Oct 30 22:03:32.291852 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.291793    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"kube-root-ca.crt\""
Oct 30 22:03:32.292178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.292156    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-plugins-e2e-extmod-b-846656\""
Oct 30 22:03:32.304569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.304531    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 22:03:32.389395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.389355    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.389395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.389405    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.389732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.389480    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.389732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.389529    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.389877 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.389764    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:32.390044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390018    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:32.390519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390490    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.390870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390545    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.390870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390720    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.390870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390836    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:32.391591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390869    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.391591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.390927    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data\") pod \"77bfd09c-2941-477e-8e75-f06def9a9920\" (UID: \"77bfd09c-2941-477e-8e75-f06def9a9920\") "
Oct 30 22:03:32.391591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.391276    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.391591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.391438    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.391591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.391470    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfs5\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.394598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.391696    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.394598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.392146    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.394598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.392171    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.394598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.392192    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bfd09c-2941-477e-8e75-f06def9a9920-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.394598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.392539    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:32.394598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.392865    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:32.394995 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.394847    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:32.396034 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.395997    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:32.397117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.397073    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:32.397451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.397426    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm" pod "77bfd09c-2941-477e-8e75-f06def9a9920" (UID: "77bfd09c-2941-477e-8e75-f06def9a9920"). InnerVolumeSpecName "kube-api-access-v75gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:32.495077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495018    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.495338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495177    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.495338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495207    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfs5\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495349    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.495375    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-846656/agc-cred-f060d103b140: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.495403    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.495491    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials podName:48f44573-3694-4043-9ef6-55e9b17ad3c6 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:32.995465181 +0000 UTC m=+3195.917923475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6") : secret "agc-cred-f060d103b140" not found
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495535    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bfd09c-2941-477e-8e75-f06def9a9920-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495553    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495569    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75gm\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-kube-api-access-v75gm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495585    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495615    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bfd09c-2941-477e-8e75-f06def9a9920-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.495646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.495629    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bfd09c-2941-477e-8e75-f06def9a9920-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:32.496668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.496630    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.496811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.496786    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.515431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.515389    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfs5\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:32.567894 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.567847    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" event={"ID":"c586f783-d197-4129-9d8f-6cc5cca6aaab","Type":"ContainerStarted","Data":"dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081"}
Oct 30 22:03:32.568131 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.568104    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:32.568203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.568168    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" event={"ID":"c586f783-d197-4129-9d8f-6cc5cca6aaab","Type":"ContainerStarted","Data":"c866cdf5db90db3d5bc8ce17e8eae5f80e94f2b274408288dd21c0fa1e0b6507"}
Oct 30 22:03:32.570433 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.570396    6695 generic.go:356] "Generic (PLEG): container finished" podID="77bfd09c-2941-477e-8e75-f06def9a9920" containerID="6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e" exitCode=0
Oct 30 22:03:32.570643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.570449    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" event={"ID":"77bfd09c-2941-477e-8e75-f06def9a9920","Type":"ContainerDied","Data":"6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e"}
Oct 30 22:03:32.570643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.570460    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"
Oct 30 22:03:32.570643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.570504    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6" event={"ID":"77bfd09c-2941-477e-8e75-f06def9a9920","Type":"ContainerDied","Data":"00ede815bcca3f80cffb141123019253a57a273181c28988b1621ee87cf4584a"}
Oct 30 22:03:32.570643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.570535    6695 scope.go:122] "RemoveContainer" containerID="6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e"
Oct 30 22:03:32.587886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.587851    6695 scope.go:122] "RemoveContainer" containerID="d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"
Oct 30 22:03:32.603863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.603624    6695 scope.go:122] "RemoveContainer" containerID="6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e"
Oct 30 22:03:32.604116 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.604093    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e\": container with ID starting with 6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e not found: ID does not exist" containerID="6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e"
Oct 30 22:03:32.604188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.604127    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e"} err="failed to get container status \"6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e\": rpc error: code = NotFound desc = could not find container \"6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e\": container with ID starting with 6348c73d1836ba3b51c541078b320c353ac2632deba7f70ea6c14242a7a2c00e not found: ID does not exist"
Oct 30 22:03:32.604188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.604147    6695 scope.go:122] "RemoveContainer" containerID="d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"
Oct 30 22:03:32.604482 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.604461    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a\": container with ID starting with d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a not found: ID does not exist" containerID="d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"
Oct 30 22:03:32.604546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.604487    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a"} err="failed to get container status \"d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a\": rpc error: code = NotFound desc = could not find container \"d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a\": container with ID starting with d92f7b179dc22c1eb1a2c6837b43cd4b355832b83396d58662b7e9f3f3c1319a not found: ID does not exist"
Oct 30 22:03:32.608518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.608449    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" podStartSLOduration=1.608429481 podStartE2EDuration="1.608429481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:32.606404006 +0000 UTC m=+3195.528862318" watchObservedRunningTime="2026-10-30 22:03:32.608429481 +0000 UTC m=+3195.530887780"
Oct 30 22:03:32.652902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.652771    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"]
Oct 30 22:03:32.656754 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.656707    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-855m6"]
Oct 30 22:03:32.699781 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.699739    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c19461/agc-cred-010bb35e298d: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:32.699781 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.699773    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:32.700040 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:32.699829    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:33.19981237 +0000 UTC m=+3196.122270678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:32.735919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:32.735875    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"]
Oct 30 22:03:33.004242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:33.004151    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:33.004417 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.004306    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-846656/agc-cred-f060d103b140: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:33.004417 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.004326    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:33.004417 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.004382    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials podName:48f44573-3694-4043-9ef6-55e9b17ad3c6 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:34.004367215 +0000 UTC m=+3196.926825492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6") : secret "agc-cred-f060d103b140" not found
Oct 30 22:03:33.214922 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.214884    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c19461/agc-cred-010bb35e298d: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:33.221629 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.215182    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:33.222024 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.221984    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:34.221947862 +0000 UTC m=+3197.144406161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:33.233435 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:33.233395    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" path="/var/lib/kubelet/pods/77bfd09c-2941-477e-8e75-f06def9a9920/volumes"
Oct 30 22:03:33.584756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:33.584708    6695 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" secret="" err="secret \"payload-processing-external-model-e2e--f2794c3d-dockercfg-pxsz6\" not found"
Oct 30 22:03:33.723205 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.723160    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-plugins-e2e-extmod-a-c19461: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:33.723431 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:33.723243    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:34.223226937 +0000 UTC m=+3197.145685217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:34.042735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.042688    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:34.042945 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.042869    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-846656/agc-cred-f060d103b140: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:34.042945 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.042899    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:34.043044 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.042982    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials podName:48f44573-3694-4043-9ef6-55e9b17ad3c6 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:36.0429549 +0000 UTC m=+3198.965413183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6") : secret "agc-cred-f060d103b140" not found
Oct 30 22:03:34.156613 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.156556    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"]
Oct 30 22:03:34.160015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.159982    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:34.189849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.189804    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"]
Oct 30 22:03:34.190915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.190883    6695 cpu_manager.go:414] "containerMap: removing container" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy"
Oct 30 22:03:34.190915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.190912    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy"
Oct 30 22:03:34.191163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.190959    6695 cpu_manager.go:414] "containerMap: removing container" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy"
Oct 30 22:03:34.191163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.190968    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy"
Oct 30 22:03:34.191163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.191149    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy"
Oct 30 22:03:34.196981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.196944    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.219557 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.219514    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"]
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.246441    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.246554    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c19461/agc-cred-010bb35e298d: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.246582    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.246655    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:36.246636232 +0000 UTC m=+3199.169094525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.246704    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.246777    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.247000    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699df\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.247120    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-plugins-e2e-extmod-a-c19461: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:34.252893 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.247183    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:35.247165576 +0000 UTC m=+3198.169623866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:34.384679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.353695    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.384679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.353931    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699df\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.384679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.354129    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.384679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.354273    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.384679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.355373    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.384679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.384031    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.392827 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.392749    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.431629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.427436    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-699df\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df\") pod \"payload-processing-external-model-6cb6ff46b5-mdhr6\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.497282 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.497226    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf" podUID="ce0c1245-a8a4-464e-a9a4-e808d2919250" containerName="payload-processing" containerID="cri-o://a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2" gracePeriod=25
Oct 30 22:03:34.516165 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.516105    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:34.778293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.778248    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"]
Oct 30 22:03:34.778915 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:34.778882    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4" podUID="48f44573-3694-4043-9ef6-55e9b17ad3c6"
Oct 30 22:03:34.845120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.842758    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"]
Oct 30 22:03:34.864452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.864410    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:34.969397 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.969342    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume\") pod \"ce0c1245-a8a4-464e-a9a4-e808d2919250\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") "
Oct 30 22:03:34.969584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.969420    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9\") pod \"ce0c1245-a8a4-464e-a9a4-e808d2919250\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") "
Oct 30 22:03:34.969738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.969595    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume\") pod \"ce0c1245-a8a4-464e-a9a4-e808d2919250\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") "
Oct 30 22:03:34.969812 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.969747    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials\") pod \"ce0c1245-a8a4-464e-a9a4-e808d2919250\" (UID: \"ce0c1245-a8a4-464e-a9a4-e808d2919250\") "
Oct 30 22:03:34.969895 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.969858    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume" pod "ce0c1245-a8a4-464e-a9a4-e808d2919250" (UID: "ce0c1245-a8a4-464e-a9a4-e808d2919250"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:34.970085 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.970062    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume" pod "ce0c1245-a8a4-464e-a9a4-e808d2919250" (UID: "ce0c1245-a8a4-464e-a9a4-e808d2919250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:34.970472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.970453    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:34.970472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.970472    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce0c1245-a8a4-464e-a9a4-e808d2919250-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:34.972225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.972190    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials" pod "ce0c1245-a8a4-464e-a9a4-e808d2919250" (UID: "ce0c1245-a8a4-464e-a9a4-e808d2919250"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:34.972513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:34.972489    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9" pod "ce0c1245-a8a4-464e-a9a4-e808d2919250" (UID: "ce0c1245-a8a4-464e-a9a4-e808d2919250"). InnerVolumeSpecName "kube-api-access-tssz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:35.072263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.072216    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tssz9\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-kube-api-access-tssz9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:35.072263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.072256    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce0c1245-a8a4-464e-a9a4-e808d2919250-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:35.290887 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:35.283514    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-plugins-e2e-extmod-a-c19461: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:35.290887 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:35.283623    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:37.283579589 +0000 UTC m=+3200.206037888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:35.613972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.613829    6695 generic.go:356] "Generic (PLEG): container finished" podID="ce0c1245-a8a4-464e-a9a4-e808d2919250" containerID="a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2" exitCode=0
Oct 30 22:03:35.613972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.613882    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"
Oct 30 22:03:35.613972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.613904    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf" event={"ID":"ce0c1245-a8a4-464e-a9a4-e808d2919250","Type":"ContainerDied","Data":"a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2"}
Oct 30 22:03:35.613972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.613967    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf" event={"ID":"ce0c1245-a8a4-464e-a9a4-e808d2919250","Type":"ContainerDied","Data":"4761be30b2ddb2daf81fb1bdb72b904cc8591f2bd13e6c85496b983a6464d086"}
Oct 30 22:03:35.614293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.613991    6695 scope.go:122] "RemoveContainer" containerID="a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2"
Oct 30 22:03:35.616366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.616283    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:35.616854 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.616740    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6" event={"ID":"0cc70851-5c8f-4b8b-a01c-db9fd55cd968","Type":"ContainerStarted","Data":"0b54076abe81b15e9c75a36b2a3f5ec88ed4ee7225ed4a7706b572868de56d4b"}
Oct 30 22:03:35.616854 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.616802    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6" event={"ID":"0cc70851-5c8f-4b8b-a01c-db9fd55cd968","Type":"ContainerStarted","Data":"683e864b27fd89e2386fc23c065e51ba80767130b4716031a7f4c9a68a865643"}
Oct 30 22:03:35.617098 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.616901    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:35.627418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.627376    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:35.628027 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.628011    6695 scope.go:122] "RemoveContainer" containerID="a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2"
Oct 30 22:03:35.628415 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:35.628394    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2\": container with ID starting with a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2 not found: ID does not exist" containerID="a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2"
Oct 30 22:03:35.628485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.628429    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2"} err="failed to get container status \"a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2\": rpc error: code = NotFound desc = could not find container \"a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2\": container with ID starting with a79fb03c12b04f973e8c0bdf5c1a875ad2f8ca5bee1d401e92fbbdae26c4bef2 not found: ID does not exist"
Oct 30 22:03:35.689872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.689831    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume\") pod \"48f44573-3694-4043-9ef6-55e9b17ad3c6\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") "
Oct 30 22:03:35.690103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.690021    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5\") pod \"48f44573-3694-4043-9ef6-55e9b17ad3c6\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") "
Oct 30 22:03:35.690103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.690065    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume\") pod \"48f44573-3694-4043-9ef6-55e9b17ad3c6\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") "
Oct 30 22:03:35.690314 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.690280    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume" pod "48f44573-3694-4043-9ef6-55e9b17ad3c6" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:35.690526 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.690504    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume" pod "48f44573-3694-4043-9ef6-55e9b17ad3c6" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:35.690999 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.690979    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:35.691116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.691004    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f44573-3694-4043-9ef6-55e9b17ad3c6-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:35.692429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.692404    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5" pod "48f44573-3694-4043-9ef6-55e9b17ad3c6" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6"). InnerVolumeSpecName "kube-api-access-dlfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:35.707101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.706692    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6" podStartSLOduration=1.70667206 podStartE2EDuration="1.70667206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:35.704828674 +0000 UTC m=+3198.627286973" watchObservedRunningTime="2026-10-30 22:03:35.70667206 +0000 UTC m=+3198.629130366"
Oct 30 22:03:35.742314 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.742271    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"]
Oct 30 22:03:35.750637 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.750074    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-6xndf"]
Oct 30 22:03:35.793271 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:35.793222    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfs5\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-kube-api-access-dlfs5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:36.104218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:36.104170    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4\" (UID: \"48f44573-3694-4043-9ef6-55e9b17ad3c6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:36.104481 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:36.104360    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-846656/agc-cred-f060d103b140: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:36.104481 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:36.104392    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4: secret "agc-cred-f060d103b140" not found
Oct 30 22:03:36.104481 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:36.104474    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials podName:48f44573-3694-4043-9ef6-55e9b17ad3c6 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:40.104451544 +0000 UTC m=+3203.026909836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4" (UID: "48f44573-3694-4043-9ef6-55e9b17ad3c6") : secret "agc-cred-f060d103b140" not found
Oct 30 22:03:36.308545 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:36.308509    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c19461/agc-cred-010bb35e298d: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:36.308545 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:36.308542    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92: secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:36.308839 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:36.308643    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:40.308621563 +0000 UTC m=+3203.231079853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : secret "agc-cred-010bb35e298d" not found
Oct 30 22:03:36.626577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:36.626537    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"
Oct 30 22:03:36.703756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:36.703696    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"]
Oct 30 22:03:36.710385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:36.710327    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-7566z5jt4"]
Oct 30 22:03:36.818064 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:36.818017    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48f44573-3694-4043-9ef6-55e9b17ad3c6-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:37.216795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:37.216748    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f44573-3694-4043-9ef6-55e9b17ad3c6" path="/var/lib/kubelet/pods/48f44573-3694-4043-9ef6-55e9b17ad3c6/volumes"
Oct 30 22:03:37.217302 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:37.217276    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0c1245-a8a4-464e-a9a4-e808d2919250" path="/var/lib/kubelet/pods/ce0c1245-a8a4-464e-a9a4-e808d2919250/volumes"
Oct 30 22:03:37.324873 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:37.324816    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-plugins-e2e-extmod-a-c19461: configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:37.325124 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:37.324910    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume podName:c586f783-d197-4129-9d8f-6cc5cca6aaab nodeName:}" failed. No retries permitted until 2026-10-30 22:03:41.32488912 +0000 UTC m=+3204.247347401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-c19461" not found
Oct 30 22:03:39.166880 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.166670    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7" podUID="885099a0-6a17-4214-b132-a57b5c1d225a" containerName="payload-processing" containerID="cri-o://1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3" gracePeriod=25
Oct 30 22:03:39.466005 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.465955    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:39.571911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.571875    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume\") pod \"885099a0-6a17-4214-b132-a57b5c1d225a\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") "
Oct 30 22:03:39.572117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.571925    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume\") pod \"885099a0-6a17-4214-b132-a57b5c1d225a\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") "
Oct 30 22:03:39.572117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.572011    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj\") pod \"885099a0-6a17-4214-b132-a57b5c1d225a\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") "
Oct 30 22:03:39.572117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.572114    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials\") pod \"885099a0-6a17-4214-b132-a57b5c1d225a\" (UID: \"885099a0-6a17-4214-b132-a57b5c1d225a\") "
Oct 30 22:03:39.572408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.572373    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume" pod "885099a0-6a17-4214-b132-a57b5c1d225a" (UID: "885099a0-6a17-4214-b132-a57b5c1d225a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:39.572408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.572390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume" pod "885099a0-6a17-4214-b132-a57b5c1d225a" (UID: "885099a0-6a17-4214-b132-a57b5c1d225a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:39.572963 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.572938    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:39.573076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.572963    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885099a0-6a17-4214-b132-a57b5c1d225a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:39.574457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.574435    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials" pod "885099a0-6a17-4214-b132-a57b5c1d225a" (UID: "885099a0-6a17-4214-b132-a57b5c1d225a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:39.574539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.574467    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj" pod "885099a0-6a17-4214-b132-a57b5c1d225a" (UID: "885099a0-6a17-4214-b132-a57b5c1d225a"). InnerVolumeSpecName "kube-api-access-t4chj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:39.601879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.601830    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" podUID="c586f783-d197-4129-9d8f-6cc5cca6aaab" containerName="payload-processing" containerID="cri-o://dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081" gracePeriod=25
Oct 30 22:03:39.653257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.653219    6695 generic.go:356] "Generic (PLEG): container finished" podID="885099a0-6a17-4214-b132-a57b5c1d225a" containerID="1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3" exitCode=0
Oct 30 22:03:39.653461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.653272    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"
Oct 30 22:03:39.653461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.653298    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7" event={"ID":"885099a0-6a17-4214-b132-a57b5c1d225a","Type":"ContainerDied","Data":"1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3"}
Oct 30 22:03:39.653461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.653355    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7" event={"ID":"885099a0-6a17-4214-b132-a57b5c1d225a","Type":"ContainerDied","Data":"ffdf2b85973cf0109de7dc15decbe9dcb27aecc7a5b7a27ee570d6e214bc1dc2"}
Oct 30 22:03:39.653461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.653379    6695 scope.go:122] "RemoveContainer" containerID="1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3"
Oct 30 22:03:39.674395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.674366    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4chj\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-kube-api-access-t4chj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:39.674541 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.674411    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/885099a0-6a17-4214-b132-a57b5c1d225a-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:39.719676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.717770    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"]
Oct 30 22:03:39.722924 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.722895    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-2w8s7"]
Oct 30 22:03:39.723849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.723828    6695 scope.go:122] "RemoveContainer" containerID="1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3"
Oct 30 22:03:39.724217 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:39.724193    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3\": container with ID starting with 1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3 not found: ID does not exist" containerID="1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3"
Oct 30 22:03:39.724285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.724228    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3"} err="failed to get container status \"1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3\": rpc error: code = NotFound desc = could not find container \"1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3\": container with ID starting with 1bae8957a701e32c46b49c9c08ed0655f1e85bdec070e78f1f6cc8c3119275c3 not found: ID does not exist"
Oct 30 22:03:39.954090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.954062    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"]
Oct 30 22:03:39.955246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955209    6695 cpu_manager.go:414] "containerMap: removing container" podUID="885099a0-6a17-4214-b132-a57b5c1d225a" containerName="payload-processing"
Oct 30 22:03:39.955246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955244    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885099a0-6a17-4214-b132-a57b5c1d225a" containerName="payload-processing"
Oct 30 22:03:39.955437 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955269    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ce0c1245-a8a4-464e-a9a4-e808d2919250" containerName="payload-processing"
Oct 30 22:03:39.955437 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955276    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0c1245-a8a4-464e-a9a4-e808d2919250" containerName="payload-processing"
Oct 30 22:03:39.955613 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955585    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="885099a0-6a17-4214-b132-a57b5c1d225a" containerName="payload-processing"
Oct 30 22:03:39.955680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955634    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0c1245-a8a4-464e-a9a4-e808d2919250" containerName="payload-processing"
Oct 30 22:03:39.955680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.955650    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="77bfd09c-2941-477e-8e75-f06def9a9920" containerName="istio-proxy"
Oct 30 22:03:39.957036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.956728    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:39.959904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.959881    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:39.993338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:39.993301    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"]
Oct 30 22:03:40.023671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.021100    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8d53e0-openshift-default-dockercfg-sqljl\""
Oct 30 22:03:40.087395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.087357    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw\") pod \"c586f783-d197-4129-9d8f-6cc5cca6aaab\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") "
Oct 30 22:03:40.087597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.087499    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume\") pod \"c586f783-d197-4129-9d8f-6cc5cca6aaab\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") "
Oct 30 22:03:40.087597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.087550    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume\") pod \"c586f783-d197-4129-9d8f-6cc5cca6aaab\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") "
Oct 30 22:03:40.087597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.087690    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials\") pod \"c586f783-d197-4129-9d8f-6cc5cca6aaab\" (UID: \"c586f783-d197-4129-9d8f-6cc5cca6aaab\") "
Oct 30 22:03:40.087944 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.087911    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume" pod "c586f783-d197-4129-9d8f-6cc5cca6aaab" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.087983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.087957    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume" pod "c586f783-d197-4129-9d8f-6cc5cca6aaab" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.088115 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088093    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088147    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088259    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088426 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088406    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088500    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088642    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088741    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvt9d\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088846    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.088964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.088907    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.089101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.089082    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.089196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.089114    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c586f783-d197-4129-9d8f-6cc5cca6aaab-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.089867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.089837    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw" pod "c586f783-d197-4129-9d8f-6cc5cca6aaab" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab"). InnerVolumeSpecName "kube-api-access-bsgtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.089966 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.089918    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials" pod "c586f783-d197-4129-9d8f-6cc5cca6aaab" (UID: "c586f783-d197-4129-9d8f-6cc5cca6aaab"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.216634    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.216705    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.216753    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.216897    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.216955    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217019    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217067    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvt9d\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217152    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217201    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217319    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217339    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsgtw\" (UniqueName: \"kubernetes.io/projected/c586f783-d197-4129-9d8f-6cc5cca6aaab-kube-api-access-bsgtw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.217803    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.219208    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.231382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.231234    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.270087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.231514    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.270087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.246037    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.360897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.359388    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvt9d\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.441625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.441181    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.467624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.458289    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.489542 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.489433    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo\") pod \"e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") " pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.562647 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.561346    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"]
Oct 30 22:03:40.563836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.563811    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c586f783-d197-4129-9d8f-6cc5cca6aaab" containerName="payload-processing"
Oct 30 22:03:40.564023 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.564005    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c586f783-d197-4129-9d8f-6cc5cca6aaab" containerName="payload-processing"
Oct 30 22:03:40.564355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.564342    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c586f783-d197-4129-9d8f-6cc5cca6aaab" containerName="payload-processing"
Oct 30 22:03:40.599266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.599237    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.602138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.602109    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"]
Oct 30 22:03:40.611376 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.611126    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:40.622199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.615227    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:40.622199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.616017    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:03:40.622199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.616182    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:40.622199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.616270    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:40.676669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.675758    6695 generic.go:356] "Generic (PLEG): container finished" podID="c586f783-d197-4129-9d8f-6cc5cca6aaab" containerID="dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081" exitCode=0
Oct 30 22:03:40.676669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.675796    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" event={"ID":"c586f783-d197-4129-9d8f-6cc5cca6aaab","Type":"ContainerDied","Data":"dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081"}
Oct 30 22:03:40.676669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.675831    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92" event={"ID":"c586f783-d197-4129-9d8f-6cc5cca6aaab","Type":"ContainerDied","Data":"c866cdf5db90db3d5bc8ce17e8eae5f80e94f2b274408288dd21c0fa1e0b6507"}
Oct 30 22:03:40.676669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.675852    6695 scope.go:122] "RemoveContainer" containerID="dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081"
Oct 30 22:03:40.676669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.675985    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"
Oct 30 22:03:40.692534 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.692512    6695 scope.go:122] "RemoveContainer" containerID="dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081"
Oct 30 22:03:40.692891 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:40.692855    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081\": container with ID starting with dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081 not found: ID does not exist" containerID="dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081"
Oct 30 22:03:40.692973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.692905    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081"} err="failed to get container status \"dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081\": rpc error: code = NotFound desc = could not find container \"dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081\": container with ID starting with dedd0a723b17c3ac3c5e7fb7964f3161cf49df61c69fc999788b8ee908558081 not found: ID does not exist"
Oct 30 22:03:40.730298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.729477    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"]
Oct 30 22:03:40.735702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.732251    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-6f848cs92"]
Oct 30 22:03:40.738580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.735942    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbjs\" (UniqueName: \"kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.738580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.736113    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.738580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.736192    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.837772 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.837737    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.837934 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.837922    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbjs\" (UniqueName: \"kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.838005 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.837993    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.838731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.838701    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.840324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.840298    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.858412 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.858388    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbjs\" (UniqueName: \"kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs\") pod \"maas-api-revoke-sub-2a12d005-v4zvh\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:40.936407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:40.936370    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:41.030802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.011861    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"]
Oct 30 22:03:41.030802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.022271    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:41.030802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.022355    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:41.030802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.022400    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:03:41.216300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.216267    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885099a0-6a17-4214-b132-a57b5c1d225a" path="/var/lib/kubelet/pods/885099a0-6a17-4214-b132-a57b5c1d225a/volumes"
Oct 30 22:03:41.216740 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.216726    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c586f783-d197-4129-9d8f-6cc5cca6aaab" path="/var/lib/kubelet/pods/c586f783-d197-4129-9d8f-6cc5cca6aaab/volumes"
Oct 30 22:03:41.251097 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.251064    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"]
Oct 30 22:03:41.254198 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:41.254167    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37995d5_3c94_4516_ae8c_6b57951041ed.slice/crio-bd7b86736f09bbdc178c5f35471e797573714b6ee69fb6f1101b6d93c74c3baf WatchSource:0}: Error finding container bd7b86736f09bbdc178c5f35471e797573714b6ee69fb6f1101b6d93c74c3baf: Status 404 returned error can't find the container with id bd7b86736f09bbdc178c5f35471e797573714b6ee69fb6f1101b6d93c74c3baf
Oct 30 22:03:41.686506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.686417    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" event={"ID":"49803228-087d-4b4c-8d79-fcc236abe8f8","Type":"ContainerStarted","Data":"1be1b9ec6a125ff27eabbc28741dfe24fa85b33b13a8de6a4f7a02e5c69c7c6a"}
Oct 30 22:03:41.686506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.686476    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" event={"ID":"49803228-087d-4b4c-8d79-fcc236abe8f8","Type":"ContainerStarted","Data":"f95b1dccc2811681dd39009de9b137fae83d8aa4a94cc91bd86387391a7cbe1a"}
Oct 30 22:03:41.689327 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.689299    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh" event={"ID":"a37995d5-3c94-4516-ae8c-6b57951041ed","Type":"ContainerStarted","Data":"c1a8b695b3a9568eb11413687b41455028997ade6eb07193c333134036805211"}
Oct 30 22:03:41.689465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.689346    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh" event={"ID":"a37995d5-3c94-4516-ae8c-6b57951041ed","Type":"ContainerStarted","Data":"bd7b86736f09bbdc178c5f35471e797573714b6ee69fb6f1101b6d93c74c3baf"}
Oct 30 22:03:41.692974 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.692950    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"]
Oct 30 22:03:41.695460 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.695439    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:41.729927 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.729874    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podStartSLOduration=2.7298522910000003 podStartE2EDuration="2.729852291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:41.727060077 +0000 UTC m=+3204.649518376" watchObservedRunningTime="2026-10-30 22:03:41.729852291 +0000 UTC m=+3204.652310589"
Oct 30 22:03:41.769977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:41.769926    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh" podStartSLOduration=1.769909663 podStartE2EDuration="1.769909663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:41.765893798 +0000 UTC m=+3204.688352097" watchObservedRunningTime="2026-10-30 22:03:41.769909663 +0000 UTC m=+3204.692367979"
Oct 30 22:03:42.240353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.240322    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"]
Oct 30 22:03:42.245071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.245046    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.263413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.263382    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"]
Oct 30 22:03:42.362787 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.362752    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.362979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.362814    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwxr\" (UniqueName: \"kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.363292 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.363203    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.465330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.465232    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.465506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.465336    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.465506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.465387    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwxr\" (UniqueName: \"kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.466038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.466008    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.467888 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.467871    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.483915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.483891    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwxr\" (UniqueName: \"kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr\") pod \"maas-api-revoke-sub-21ed5937-6tkpn\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.597805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.597548    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:42.612517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.612491    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:42.619955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.619915    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:42.701876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.701839    6695 generic.go:356] "Generic (PLEG): container finished" podID="a37995d5-3c94-4516-ae8c-6b57951041ed" containerID="c1a8b695b3a9568eb11413687b41455028997ade6eb07193c333134036805211" exitCode=0
Oct 30 22:03:42.702074 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.701922    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh" event={"ID":"a37995d5-3c94-4516-ae8c-6b57951041ed","Type":"ContainerDied","Data":"c1a8b695b3a9568eb11413687b41455028997ade6eb07193c333134036805211"}
Oct 30 22:03:42.702348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.702331    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:42.703504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.703482    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:03:42.906318 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:42.906289    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"]
Oct 30 22:03:42.908736 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:42.908706    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ac4b1b_8751_402e_8b17_b2fda048e437.slice/crio-15c43683acb9934d2f4740c503b5563e2988243e9cb64d000f13e72c23c0d014 WatchSource:0}: Error finding container 15c43683acb9934d2f4740c503b5563e2988243e9cb64d000f13e72c23c0d014: Status 404 returned error can't find the container with id 15c43683acb9934d2f4740c503b5563e2988243e9cb64d000f13e72c23c0d014
Oct 30 22:03:43.712443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.712331    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn" event={"ID":"71ac4b1b-8751-402e-8b17-b2fda048e437","Type":"ContainerStarted","Data":"0aed215f8573243807e174a46572f308bf2dd6eb4822ceb055fa126c9353fd16"}
Oct 30 22:03:43.712443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.712408    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn" event={"ID":"71ac4b1b-8751-402e-8b17-b2fda048e437","Type":"ContainerStarted","Data":"15c43683acb9934d2f4740c503b5563e2988243e9cb64d000f13e72c23c0d014"}
Oct 30 22:03:43.759418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.759352    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn" podStartSLOduration=1.759332755 podStartE2EDuration="1.759332755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:43.755799341 +0000 UTC m=+3206.678257638" watchObservedRunningTime="2026-10-30 22:03:43.759332755 +0000 UTC m=+3206.681791053"
Oct 30 22:03:43.916019 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.915987    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:43.992672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.990636    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca\") pod \"a37995d5-3c94-4516-ae8c-6b57951041ed\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") "
Oct 30 22:03:43.992672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.990865    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs\" (UniqueName: \"kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs\") pod \"a37995d5-3c94-4516-ae8c-6b57951041ed\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") "
Oct 30 22:03:43.992672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.990985    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca\") pod \"a37995d5-3c94-4516-ae8c-6b57951041ed\" (UID: \"a37995d5-3c94-4516-ae8c-6b57951041ed\") "
Oct 30 22:03:43.992672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.991729    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca" pod "a37995d5-3c94-4516-ae8c-6b57951041ed" (UID: "a37995d5-3c94-4516-ae8c-6b57951041ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:43.992672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.991909    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:43.997987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.997434    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca" pod "a37995d5-3c94-4516-ae8c-6b57951041ed" (UID: "a37995d5-3c94-4516-ae8c-6b57951041ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:43.997987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:43.997572    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs" pod "a37995d5-3c94-4516-ae8c-6b57951041ed" (UID: "a37995d5-3c94-4516-ae8c-6b57951041ed"). InnerVolumeSpecName "kube-api-access-2cbjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.093788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.093755    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbjs\" (UniqueName: \"kubernetes.io/projected/a37995d5-3c94-4516-ae8c-6b57951041ed-kube-api-access-2cbjs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:44.093788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.093788    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37995d5-3c94-4516-ae8c-6b57951041ed-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:44.793203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.793165    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh" event={"ID":"a37995d5-3c94-4516-ae8c-6b57951041ed","Type":"ContainerDied","Data":"bd7b86736f09bbdc178c5f35471e797573714b6ee69fb6f1101b6d93c74c3baf"}
Oct 30 22:03:44.793804 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.793782    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7b86736f09bbdc178c5f35471e797573714b6ee69fb6f1101b6d93c74c3baf"
Oct 30 22:03:44.807624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.798823    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"
Oct 30 22:03:44.816228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.815496    6695 generic.go:356] "Generic (PLEG): container finished" podID="71ac4b1b-8751-402e-8b17-b2fda048e437" containerID="0aed215f8573243807e174a46572f308bf2dd6eb4822ceb055fa126c9353fd16" exitCode=0
Oct 30 22:03:44.818776 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:44.817872    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn" event={"ID":"71ac4b1b-8751-402e-8b17-b2fda048e437","Type":"ContainerDied","Data":"0aed215f8573243807e174a46572f308bf2dd6eb4822ceb055fa126c9353fd16"}
Oct 30 22:03:45.346929 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:45.346897    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b35fff36b219: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:45.347163 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:45.347147    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:45.347697 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:45.347678    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials podName:091d14b6-6cb4-4a88-b570-627764ce3c70 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:45.847656253 +0000 UTC m=+3208.770114549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials") pod "payload-processing-external-model-7d8545fb8d-stcz7" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70") : secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:45.849868 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:45.849376    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b35fff36b219: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:45.849868 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:45.849418    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:45.849868 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:45.849478    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials podName:091d14b6-6cb4-4a88-b570-627764ce3c70 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:46.849459026 +0000 UTC m=+3209.771917305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials") pod "payload-processing-external-model-7d8545fb8d-stcz7" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70") : secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:46.120847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.120808    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:46.230750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.230404    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"]
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.232699    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71ac4b1b-8751-402e-8b17-b2fda048e437" containerName="revoke-keys"
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.232726    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ac4b1b-8751-402e-8b17-b2fda048e437" containerName="revoke-keys"
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.232776    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a37995d5-3c94-4516-ae8c-6b57951041ed" containerName="revoke-keys"
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.232785    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37995d5-3c94-4516-ae8c-6b57951041ed" containerName="revoke-keys"
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.234513    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ac4b1b-8751-402e-8b17-b2fda048e437" containerName="revoke-keys"
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.234546    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37995d5-3c94-4516-ae8c-6b57951041ed" containerName="revoke-keys"
Oct 30 22:03:46.245228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.241978    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.276622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.272969    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8d53e0\""
Oct 30 22:03:46.290920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.284865    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca\") pod \"71ac4b1b-8751-402e-8b17-b2fda048e437\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") "
Oct 30 22:03:46.290920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.285077    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr\" (UniqueName: \"kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr\") pod \"71ac4b1b-8751-402e-8b17-b2fda048e437\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") "
Oct 30 22:03:46.290920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.285213    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca\") pod \"71ac4b1b-8751-402e-8b17-b2fda048e437\" (UID: \"71ac4b1b-8751-402e-8b17-b2fda048e437\") "
Oct 30 22:03:46.292719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.291323    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca" pod "71ac4b1b-8751-402e-8b17-b2fda048e437" (UID: "71ac4b1b-8751-402e-8b17-b2fda048e437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:46.293943 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.293918    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"]
Oct 30 22:03:46.386543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.386454    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr" pod "71ac4b1b-8751-402e-8b17-b2fda048e437" (UID: "71ac4b1b-8751-402e-8b17-b2fda048e437"). InnerVolumeSpecName "kube-api-access-xpwxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:46.393829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.393778    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.394117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.394096    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftwl\" (UniqueName: \"kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl\") pod \"payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.394377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.394359    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:46.394467 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.394382    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwxr\" (UniqueName: \"kubernetes.io/projected/71ac4b1b-8751-402e-8b17-b2fda048e437-kube-api-access-xpwxr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:46.397129 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.397103    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8d53e0-dockercfg-fkj86\""
Oct 30 22:03:46.538626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.516741    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.538626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.516993    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftwl\" (UniqueName: \"kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl\") pod \"payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.538626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.519229    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.655815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.654916    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca" pod "71ac4b1b-8751-402e-8b17-b2fda048e437" (UID: "71ac4b1b-8751-402e-8b17-b2fda048e437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:46.655815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.655312    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ac4b1b-8751-402e-8b17-b2fda048e437-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:46.661884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.661853    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftwl\" (UniqueName: \"kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl\") pod \"payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:46.700084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.699868    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6" podUID="0cc70851-5c8f-4b8b-a01c-db9fd55cd968" containerName="payload-processing" containerID="cri-o://0b54076abe81b15e9c75a36b2a3f5ec88ed4ee7225ed4a7706b572868de56d4b" gracePeriod=25
Oct 30 22:03:46.713539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.704176    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"]
Oct 30 22:03:46.804939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.804904    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:46.820916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.820867    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"]
Oct 30 22:03:46.936674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.929647    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:46.936674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.929802    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44vd\" (UniqueName: \"kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd\") pod \"payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:46.936674 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:46.930009    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b35fff36b219: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:46.936674 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:46.930035    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:46.936674 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:46.930099    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials podName:091d14b6-6cb4-4a88-b570-627764ce3c70 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:48.930082586 +0000 UTC m=+3211.852540874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials") pod "payload-processing-external-model-7d8545fb8d-stcz7" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70") : secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:46.936674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:46.930640    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:47.032184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.032155    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44vd\" (UniqueName: \"kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd\") pod \"payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:47.049626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.032688    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:47.050022 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.032944    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"
Oct 30 22:03:47.050781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.032961    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn" event={"ID":"71ac4b1b-8751-402e-8b17-b2fda048e437","Type":"ContainerDied","Data":"15c43683acb9934d2f4740c503b5563e2988243e9cb64d000f13e72c23c0d014"}
Oct 30 22:03:47.050951 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.050934    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15c43683acb9934d2f4740c503b5563e2988243e9cb64d000f13e72c23c0d014"
Oct 30 22:03:47.051066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.047872    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:47.116982 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.107341    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44vd\" (UniqueName: \"kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd\") pod \"payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:47.283628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.283214    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:47.920638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:47.916761    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.013625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:48.009851    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.168622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:48.168490    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.814199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:48.778091    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"]
Oct 30 22:03:48.860280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:48.838709    6695 generic.go:356] "Generic (PLEG): container finished" podID="0cc70851-5c8f-4b8b-a01c-db9fd55cd968" containerID="0b54076abe81b15e9c75a36b2a3f5ec88ed4ee7225ed4a7706b572868de56d4b" exitCode=0
Oct 30 22:03:48.860280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:48.838874    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6" event={"ID":"0cc70851-5c8f-4b8b-a01c-db9fd55cd968","Type":"ContainerDied","Data":"0b54076abe81b15e9c75a36b2a3f5ec88ed4ee7225ed4a7706b572868de56d4b"}
Oct 30 22:03:48.957860 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:48.957824    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b35fff36b219: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:48.958065 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:48.958050    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bba74b814b2d: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:48.958180 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:48.958168    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7: [secret "agc-cred-b35fff36b219" not found, secret "agc-cred-bba74b814b2d" not found]
Oct 30 22:03:48.960732 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:48.960709    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials podName:091d14b6-6cb4-4a88-b570-627764ce3c70 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:52.960683099 +0000 UTC m=+3215.883141389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials") pod "payload-processing-external-model-7d8545fb8d-stcz7" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70") : [secret "agc-cred-b35fff36b219" not found, secret "agc-cred-bba74b814b2d" not found]
Oct 30 22:03:49.001240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:48.973717    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.008174 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.008125    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 22:03:49.034626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.020828    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.071945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.071822    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 22:03:49.360507 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.360386    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:03:49.423639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.401860    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 22:03:49.507007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.506952    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:49.610690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.597949    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 22:03:50.004715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.680898    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume\") pod \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") "
Oct 30 22:03:50.004715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.003202    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume\") pod \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") "
Oct 30 22:03:50.004715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.003356    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df\") pod \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") "
Oct 30 22:03:50.004715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.003635    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials\") pod \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\" (UID: \"0cc70851-5c8f-4b8b-a01c-db9fd55cd968\") "
Oct 30 22:03:50.024681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:49.681423    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume" pod "0cc70851-5c8f-4b8b-a01c-db9fd55cd968" (UID: "0cc70851-5c8f-4b8b-a01c-db9fd55cd968"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:50.024681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.005883    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume" pod "0cc70851-5c8f-4b8b-a01c-db9fd55cd968" (UID: "0cc70851-5c8f-4b8b-a01c-db9fd55cd968"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:50.024681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.002759    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 22:03:50.024681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.006661    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 22:03:50.024681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.006881    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"]
Oct 30 22:03:50.107503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.107458    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:50.107503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.107504    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:50.112693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.111225    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials" pod "0cc70851-5c8f-4b8b-a01c-db9fd55cd968" (UID: "0cc70851-5c8f-4b8b-a01c-db9fd55cd968"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:50.174456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.174364    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df" pod "0cc70851-5c8f-4b8b-a01c-db9fd55cd968" (UID: "0cc70851-5c8f-4b8b-a01c-db9fd55cd968"). InnerVolumeSpecName "kube-api-access-699df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:50.315697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.290523    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699df\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-kube-api-access-699df\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:50.315697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.290558    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cc70851-5c8f-4b8b-a01c-db9fd55cd968-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:50.377573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.375960    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"]
Oct 30 22:03:50.378240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.377980    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0cc70851-5c8f-4b8b-a01c-db9fd55cd968" containerName="payload-processing"
Oct 30 22:03:50.378240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.378078    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc70851-5c8f-4b8b-a01c-db9fd55cd968" containerName="payload-processing"
Oct 30 22:03:50.379352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.378596    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc70851-5c8f-4b8b-a01c-db9fd55cd968" containerName="payload-processing"
Oct 30 22:03:50.385673 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.385575    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.398519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.398415    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"]
Oct 30 22:03:50.544328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.529618    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" event={"ID":"cd793a5f-8112-4109-9c13-3f393b019c13","Type":"ContainerStarted","Data":"3766f2e30c783d09307bd371456493f7f77de8c58cc2ad3b1a617b5622781171"}
Oct 30 22:03:50.547788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.545922    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.547788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.546074    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.547788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.546136    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wjg\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.547788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.546176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.588766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.588723    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"
Oct 30 22:03:50.590693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.590578    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6" event={"ID":"0cc70851-5c8f-4b8b-a01c-db9fd55cd968","Type":"ContainerDied","Data":"683e864b27fd89e2386fc23c065e51ba80767130b4716031a7f4c9a68a865643"}
Oct 30 22:03:50.591017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.590991    6695 scope.go:122] "RemoveContainer" containerID="0b54076abe81b15e9c75a36b2a3f5ec88ed4ee7225ed4a7706b572868de56d4b"
Oct 30 22:03:50.651845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.648545    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.651845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.648723    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.651845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.648780    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wjg\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.651845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.648819    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.651845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.651142    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.651845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.651537    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.653797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.653760    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.675091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.675048    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"]
Oct 30 22:03:50.686155 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.686112    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-mdhr6"]
Oct 30 22:03:50.687539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.687505    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wjg\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg\") pod \"payload-processing-external-model-696d6b866b-q5thr\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") " pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:50.889472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:50.889338    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:51.264185 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.233741    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc70851-5c8f-4b8b-a01c-db9fd55cd968" path="/var/lib/kubelet/pods/0cc70851-5c8f-4b8b-a01c-db9fd55cd968/volumes"
Oct 30 22:03:51.355927 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:51.355893    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af94d62_6007_4188_9aa6_0b7380f614f5.slice/crio-0de5fb4577a1309fa39599222e9767725b58a2ea2f17e5198ba3338a690b2d5a WatchSource:0}: Error finding container 0de5fb4577a1309fa39599222e9767725b58a2ea2f17e5198ba3338a690b2d5a: Status 404 returned error can't find the container with id 0de5fb4577a1309fa39599222e9767725b58a2ea2f17e5198ba3338a690b2d5a
Oct 30 22:03:51.385367 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.369455    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"]
Oct 30 22:03:51.608093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.602932    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" event={"ID":"cd793a5f-8112-4109-9c13-3f393b019c13","Type":"ContainerStarted","Data":"3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489"}
Oct 30 22:03:51.608093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.603006    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:03:51.608093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.605385    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" event={"ID":"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30","Type":"ContainerStarted","Data":"2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5"}
Oct 30 22:03:51.608093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.605449    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" event={"ID":"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30","Type":"ContainerStarted","Data":"e345f28d893db96d7483c8974a3b13b4524d20e9bc242d36a282d9dda1820b65"}
Oct 30 22:03:51.608093 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.605976    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:03:51.609047 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.609005    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr" event={"ID":"0af94d62-6007-4188-9aa6-0b7380f614f5","Type":"ContainerStarted","Data":"be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229"}
Oct 30 22:03:51.609195 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.609062    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr" event={"ID":"0af94d62-6007-4188-9aa6-0b7380f614f5","Type":"ContainerStarted","Data":"0de5fb4577a1309fa39599222e9767725b58a2ea2f17e5198ba3338a690b2d5a"}
Oct 30 22:03:51.609195 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.609093    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:51.646475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.646398    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" podStartSLOduration=5.646379316 podStartE2EDuration="5.646379316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:51.641205894 +0000 UTC m=+3214.563664191" watchObservedRunningTime="2026-10-30 22:03:51.646379316 +0000 UTC m=+3214.568837592"
Oct 30 22:03:51.679951 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.679859    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" podStartSLOduration=5.679845452 podStartE2EDuration="5.679845452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:51.677801 +0000 UTC m=+3214.600259298" watchObservedRunningTime="2026-10-30 22:03:51.679845452 +0000 UTC m=+3214.602303767"
Oct 30 22:03:51.721684 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:51.721615    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr" podStartSLOduration=1.721584437 podStartE2EDuration="1.721584437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:51.718387272 +0000 UTC m=+3214.640845570" watchObservedRunningTime="2026-10-30 22:03:51.721584437 +0000 UTC m=+3214.644042736"
Oct 30 22:03:52.779259 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.779224    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d9c68187ac0c: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:52.780478 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.780448    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-696d6b866b-q5thr: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:52.780783 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.780768    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials podName:0af94d62-6007-4188-9aa6-0b7380f614f5 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:53.280739646 +0000 UTC m=+3216.203197939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials") pod "payload-processing-external-model-696d6b866b-q5thr" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5") : secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:52.981906 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.981865    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b35fff36b219: secret "agc-cred-b35fff36b219" not found
Oct 30 22:03:52.981906 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.981902    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bba74b814b2d: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:52.981906 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.981909    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d9c68187ac0c: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:52.982200 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.981921    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7: [secret "agc-cred-b35fff36b219" not found, secret "agc-cred-bba74b814b2d" not found, secret "agc-cred-d9c68187ac0c" not found]
Oct 30 22:03:52.982200 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:52.981982    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials podName:091d14b6-6cb4-4a88-b570-627764ce3c70 nodeName:}" failed. No retries permitted until 2026-10-30 22:04:00.981966221 +0000 UTC m=+3223.904424496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials") pod "payload-processing-external-model-7d8545fb8d-stcz7" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70") : [secret "agc-cred-b35fff36b219" not found, secret "agc-cred-bba74b814b2d" not found, secret "agc-cred-d9c68187ac0c" not found]
Oct 30 22:03:53.314398 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:53.314338    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d9c68187ac0c: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:53.315288 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:53.315251    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-696d6b866b-q5thr: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:53.315692 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:53.315672    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials podName:0af94d62-6007-4188-9aa6-0b7380f614f5 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:54.315646697 +0000 UTC m=+3217.238104974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials") pod "payload-processing-external-model-696d6b866b-q5thr" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5") : secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:53.865636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:53.863953    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"]
Oct 30 22:03:53.887459 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:53.886285    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"]
Oct 30 22:03:54.379628 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:54.378014    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d9c68187ac0c: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:54.379628 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:54.378048    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-696d6b866b-q5thr: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:54.379628 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:54.378111    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials podName:0af94d62-6007-4188-9aa6-0b7380f614f5 nodeName:}" failed. No retries permitted until 2026-10-30 22:03:56.378090874 +0000 UTC m=+3219.300549169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials") pod "payload-processing-external-model-696d6b866b-q5thr" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5") : secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:55.714253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.714213    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"]
Oct 30 22:03:55.721053 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.721008    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.744396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.744363    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"kube-root-ca.crt\""
Oct 30 22:03:55.744793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.744772    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"agc-cred-cc975ee1bf99\""
Oct 30 22:03:55.744989 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.744975    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"openshift-service-ca.crt\""
Oct 30 22:03:55.745183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.745168    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"routing-overlay\""
Oct 30 22:03:55.745305 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.745221    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"]
Oct 30 22:03:55.745474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.745300    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdb168\""
Oct 30 22:03:55.749549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.749522    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"payload-processing-external-model-e2e--ca9775af-dockercfg-878zz\""
Oct 30 22:03:55.807211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.807164    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbdh\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.807439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.807316    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.807439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.807348    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.807614 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.807574    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.913251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.913195    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbdh\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.913697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.913667    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.913858 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.913842    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.914124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.914107    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.928640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.915274    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.930795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.930758    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.965664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.965533    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:55.965896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:55.965864    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbdh\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:56.061631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.060917    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:56.435949 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:56.435914    6695 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d9c68187ac0c: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:56.435949 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:56.435944    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-696d6b866b-q5thr: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:56.436184 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:56.436003    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials podName:0af94d62-6007-4188-9aa6-0b7380f614f5 nodeName:}" failed. No retries permitted until 2026-10-30 22:04:00.435985709 +0000 UTC m=+3223.358444004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials") pod "payload-processing-external-model-696d6b866b-q5thr" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5") : secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:56.564167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.564123    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"]
Oct 30 22:03:56.566547 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:56.566518    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee057f9_9151_482f_b782_f080deb1056a.slice/crio-0f747a618b5131a3ba96cdf4d83d7f1ec5a12317e4491453f4d3f1466ee63a0f WatchSource:0}: Error finding container 0f747a618b5131a3ba96cdf4d83d7f1ec5a12317e4491453f4d3f1466ee63a0f: Status 404 returned error can't find the container with id 0f747a618b5131a3ba96cdf4d83d7f1ec5a12317e4491453f4d3f1466ee63a0f
Oct 30 22:03:56.745686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.742337    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c" event={"ID":"8ee057f9-9151-482f-b782-f080deb1056a","Type":"ContainerStarted","Data":"0f747a618b5131a3ba96cdf4d83d7f1ec5a12317e4491453f4d3f1466ee63a0f"}
Oct 30 22:03:56.783090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.783046    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"]
Oct 30 22:03:56.788688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.788657    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.807525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.807482    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"routing-overlay\""
Oct 30 22:03:56.808316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.808286    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"]
Oct 30 22:03:56.813940 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.813912    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-427a6e\""
Oct 30 22:03:56.813940 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.813935    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"kube-root-ca.crt\""
Oct 30 22:03:56.813940 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.813936    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"openshift-service-ca.crt\""
Oct 30 22:03:56.815194 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.815169    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"agc-cred-f1aaf8e664df\""
Oct 30 22:03:56.817296 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.817272    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-427a6e\"/\"payload-processing-external-model-e2e--d53c5c12-dockercfg-89tq8\""
Oct 30 22:03:56.846043 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.845908    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.846222 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.846193    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w6g\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.846422 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.846402    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.846468 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.846450    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.951627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.950691    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w6g\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.951627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.950906    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.951627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.950962    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.951627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.951137    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.958589 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.952428    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.970056 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.960262    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.970056 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.968314    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:56.987631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:56.980774    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w6g\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g\") pod \"payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp\" (UID: \"93885823-090d-4eab-9518-98282531c551\") " pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:57.105030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.104875    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:57.428124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.428097    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"]
Oct 30 22:03:57.430798 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:57.430757    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93885823_090d_4eab_9518_98282531c551.slice/crio-d2f7b245696a45673217d79df438ed2bffa6bf93d69843782093ef4f3ea2973a WatchSource:0}: Error finding container d2f7b245696a45673217d79df438ed2bffa6bf93d69843782093ef4f3ea2973a: Status 404 returned error can't find the container with id d2f7b245696a45673217d79df438ed2bffa6bf93d69843782093ef4f3ea2973a
Oct 30 22:03:57.777029 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.776767    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c" event={"ID":"8ee057f9-9151-482f-b782-f080deb1056a","Type":"ContainerStarted","Data":"3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62"}
Oct 30 22:03:57.778227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.778183    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:03:57.781651 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.781566    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp" event={"ID":"93885823-090d-4eab-9518-98282531c551","Type":"ContainerStarted","Data":"28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08"}
Oct 30 22:03:57.781867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.781689    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:03:57.781867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.781729    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp" event={"ID":"93885823-090d-4eab-9518-98282531c551","Type":"ContainerStarted","Data":"d2f7b245696a45673217d79df438ed2bffa6bf93d69843782093ef4f3ea2973a"}
Oct 30 22:03:57.818316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.818255    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c" podStartSLOduration=2.81823795 podStartE2EDuration="2.81823795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:57.816561382 +0000 UTC m=+3220.739019674" watchObservedRunningTime="2026-10-30 22:03:57.81823795 +0000 UTC m=+3220.740696241"
Oct 30 22:03:57.857304 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:57.857248    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp" podStartSLOduration=1.857231857 podStartE2EDuration="1.857231857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:57.853684221 +0000 UTC m=+3220.776142529" watchObservedRunningTime="2026-10-30 22:03:57.857231857 +0000 UTC m=+3220.779690154"
Oct 30 22:03:58.865400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:58.865267    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7" podUID="091d14b6-6cb4-4a88-b570-627764ce3c70" containerName="payload-processing" containerID="cri-o://d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde" gracePeriod=25
Oct 30 22:03:58.890036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:58.889957    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr" podUID="0af94d62-6007-4188-9aa6-0b7380f614f5" containerName="payload-processing" containerID="cri-o://be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229" gracePeriod=25
Oct 30 22:03:59.095472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.093155    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"]
Oct 30 22:03:59.106275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.106209    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.122010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.121913    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a42c31\""
Oct 30 22:03:59.122590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.122549    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"]
Oct 30 22:03:59.135036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.134976    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a42c31-dockercfg-7c6pc\""
Oct 30 22:03:59.187777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.187737    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8bp\" (UniqueName: \"kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp\") pod \"payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.188108 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.188074    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.301962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.290210    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8bp\" (UniqueName: \"kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp\") pod \"payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.301962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.290578    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.301962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.291828    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.354064 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.353771    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8bp\" (UniqueName: \"kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp\") pod \"payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.487137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.484344    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:03:59.552653 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.552616    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:03:59.560128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.560089    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624032    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume\") pod \"091d14b6-6cb4-4a88-b570-627764ce3c70\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") "
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624190    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume\") pod \"0af94d62-6007-4188-9aa6-0b7380f614f5\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") "
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624269    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk\") pod \"091d14b6-6cb4-4a88-b570-627764ce3c70\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") "
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624305    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg\") pod \"0af94d62-6007-4188-9aa6-0b7380f614f5\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") "
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624334    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume\") pod \"091d14b6-6cb4-4a88-b570-627764ce3c70\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") "
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624368    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials\") pod \"0af94d62-6007-4188-9aa6-0b7380f614f5\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") "
Oct 30 22:03:59.624627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624553    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume\") pod \"0af94d62-6007-4188-9aa6-0b7380f614f5\" (UID: \"0af94d62-6007-4188-9aa6-0b7380f614f5\") "
Oct 30 22:03:59.625200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.624641    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials\") pod \"091d14b6-6cb4-4a88-b570-627764ce3c70\" (UID: \"091d14b6-6cb4-4a88-b570-627764ce3c70\") "
Oct 30 22:03:59.645244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.645144    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg" pod "0af94d62-6007-4188-9aa6-0b7380f614f5" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5"). InnerVolumeSpecName "kube-api-access-f2wjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.645832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.645749    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume" pod "0af94d62-6007-4188-9aa6-0b7380f614f5" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.650634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.647468    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume" pod "0af94d62-6007-4188-9aa6-0b7380f614f5" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.650634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.647794    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume" pod "091d14b6-6cb4-4a88-b570-627764ce3c70" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.650634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.647938    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume" pod "091d14b6-6cb4-4a88-b570-627764ce3c70" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:59.668173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.668104    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk" pod "091d14b6-6cb4-4a88-b570-627764ce3c70" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70"). InnerVolumeSpecName "kube-api-access-kd5nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.688667 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.687821    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials" pod "0af94d62-6007-4188-9aa6-0b7380f614f5" (UID: "0af94d62-6007-4188-9aa6-0b7380f614f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.690316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.690107    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials" pod "091d14b6-6cb4-4a88-b570-627764ce3c70" (UID: "091d14b6-6cb4-4a88-b570-627764ce3c70"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727042    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727083    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727099    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727115    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/091d14b6-6cb4-4a88-b570-627764ce3c70-kube-api-access-kd5nk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727127    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2wjg\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-kube-api-access-f2wjg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727142    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091d14b6-6cb4-4a88-b570-627764ce3c70-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727153    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0af94d62-6007-4188-9aa6-0b7380f614f5-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.727189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.727164    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0af94d62-6007-4188-9aa6-0b7380f614f5-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:03:59.801518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.801480    6695 generic.go:356] "Generic (PLEG): container finished" podID="0af94d62-6007-4188-9aa6-0b7380f614f5" containerID="be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229" exitCode=0
Oct 30 22:03:59.801518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.801530    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"
Oct 30 22:03:59.801816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.801569    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr" event={"ID":"0af94d62-6007-4188-9aa6-0b7380f614f5","Type":"ContainerDied","Data":"be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229"}
Oct 30 22:03:59.801816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.801675    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696d6b866b-q5thr" event={"ID":"0af94d62-6007-4188-9aa6-0b7380f614f5","Type":"ContainerDied","Data":"0de5fb4577a1309fa39599222e9767725b58a2ea2f17e5198ba3338a690b2d5a"}
Oct 30 22:03:59.801816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.801702    6695 scope.go:122] "RemoveContainer" containerID="be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229"
Oct 30 22:03:59.803515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.803486    6695 generic.go:356] "Generic (PLEG): container finished" podID="091d14b6-6cb4-4a88-b570-627764ce3c70" containerID="d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde" exitCode=0
Oct 30 22:03:59.803700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.803540    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"
Oct 30 22:03:59.803700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.803570    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7" event={"ID":"091d14b6-6cb4-4a88-b570-627764ce3c70","Type":"ContainerDied","Data":"d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde"}
Oct 30 22:03:59.803700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.803669    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7" event={"ID":"091d14b6-6cb4-4a88-b570-627764ce3c70","Type":"ContainerDied","Data":"dae3bc780ce6b8bf47857fd3cf5a2c2a042e9ef2430d82f49774e7f08edd33e8"}
Oct 30 22:03:59.814794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.814762    6695 scope.go:122] "RemoveContainer" containerID="be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229"
Oct 30 22:03:59.815254 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:59.815225    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229\": container with ID starting with be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229 not found: ID does not exist" containerID="be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229"
Oct 30 22:03:59.815390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.815264    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229"} err="failed to get container status \"be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229\": rpc error: code = NotFound desc = could not find container \"be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229\": container with ID starting with be4c990bbffd472e71a4b896a243079ca9d684cd566bfe15b21194c64d826229 not found: ID does not exist"
Oct 30 22:03:59.815390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.815285    6695 scope.go:122] "RemoveContainer" containerID="d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde"
Oct 30 22:03:59.832528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.832408    6695 scope.go:122] "RemoveContainer" containerID="d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde"
Oct 30 22:03:59.832928 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:03:59.832898    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde\": container with ID starting with d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde not found: ID does not exist" containerID="d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde"
Oct 30 22:03:59.833049 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.832940    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde"} err="failed to get container status \"d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde\": rpc error: code = NotFound desc = could not find container \"d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde\": container with ID starting with d059aad958d789da8ada642a198ec22560036839ca675b445cc4e129a2ce6bde not found: ID does not exist"
Oct 30 22:03:59.869565 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.869530    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"]
Oct 30 22:03:59.876208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.876161    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-696d6b866b-q5thr"]
Oct 30 22:03:59.917107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.917000    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"]
Oct 30 22:03:59.921326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.921288    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d8545fb8d-stcz7"]
Oct 30 22:03:59.948098 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:03:59.948055    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"]
Oct 30 22:03:59.949456 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:03:59.949417    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89a2d5d_4cbd_41b0_bd87_e5a50e9b1e7d.slice/crio-6f8c7242cbe75934314ab8c9a873a923eae097fc143e797fdeb01a9afc8cf036 WatchSource:0}: Error finding container 6f8c7242cbe75934314ab8c9a873a923eae097fc143e797fdeb01a9afc8cf036: Status 404 returned error can't find the container with id 6f8c7242cbe75934314ab8c9a873a923eae097fc143e797fdeb01a9afc8cf036
Oct 30 22:04:00.817410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:00.817367    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" event={"ID":"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d","Type":"ContainerStarted","Data":"02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8"}
Oct 30 22:04:00.817659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:00.817435    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" event={"ID":"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d","Type":"ContainerStarted","Data":"6f8c7242cbe75934314ab8c9a873a923eae097fc143e797fdeb01a9afc8cf036"}
Oct 30 22:04:00.817659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:00.817463    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:04:00.855328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:00.855272    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" podStartSLOduration=2.855258448 podStartE2EDuration="2.855258448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:00.854550773 +0000 UTC m=+3223.777009070" watchObservedRunningTime="2026-10-30 22:04:00.855258448 +0000 UTC m=+3223.777716747"
Oct 30 22:04:01.081932 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:04:01.081833    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:04:01.217521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:01.217485    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091d14b6-6cb4-4a88-b570-627764ce3c70" path="/var/lib/kubelet/pods/091d14b6-6cb4-4a88-b570-627764ce3c70/volumes"
Oct 30 22:04:01.218044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:01.218025    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af94d62-6007-4188-9aa6-0b7380f614f5" path="/var/lib/kubelet/pods/0af94d62-6007-4188-9aa6-0b7380f614f5/volumes"
Oct 30 22:04:02.678439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:02.678410    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:04:02.679002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:02.678480    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:04:04.287529 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:04.287476    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"]
Oct 30 22:04:04.291318 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:04.291252    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:04:08.485781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.485737    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"]
Oct 30 22:04:08.486903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.486876    6695 cpu_manager.go:414] "containerMap: removing container" podUID="091d14b6-6cb4-4a88-b570-627764ce3c70" containerName="payload-processing"
Oct 30 22:04:08.486903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.486902    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091d14b6-6cb4-4a88-b570-627764ce3c70" containerName="payload-processing"
Oct 30 22:04:08.487066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.486946    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0af94d62-6007-4188-9aa6-0b7380f614f5" containerName="payload-processing"
Oct 30 22:04:08.487066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.486951    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af94d62-6007-4188-9aa6-0b7380f614f5" containerName="payload-processing"
Oct 30 22:04:08.487166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.487094    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="091d14b6-6cb4-4a88-b570-627764ce3c70" containerName="payload-processing"
Oct 30 22:04:08.487166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.487116    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af94d62-6007-4188-9aa6-0b7380f614f5" containerName="payload-processing"
Oct 30 22:04:08.494585 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.494556    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.509000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.508971    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-427a6e-openshift-default-dockercfg-kkqzj\""
Oct 30 22:04:08.524868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.524809    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"]
Oct 30 22:04:08.589828 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.589765    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590009 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.589902    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpx6\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590009 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.589946    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.590016    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.590058    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.590079    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.590227    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.590264    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.590354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.590329    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.691987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.691947    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692018    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpx6\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692047    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692102    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692155    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692179    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692320    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692374    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692436    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.692841 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.692720    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.693196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.693168    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.693196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.693193    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.693439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.693220    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.693439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.693297    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.694724 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.694703    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.694883 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.694863    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.711217 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.711177    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.711386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.711321    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpx6\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6\") pod \"e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") " pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:08.811390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:08.811355    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:09.145550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.145510    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"]
Oct 30 22:04:09.147563 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:04:09.147529    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fe97be_6faf_4aad_a571_6307e44e00e4.slice/crio-00a59afa14b6063d2c4ea9aa98c55c3e7be7d3e5a8f6d68e18eba056765100e2 WatchSource:0}: Error finding container 00a59afa14b6063d2c4ea9aa98c55c3e7be7d3e5a8f6d68e18eba056765100e2: Status 404 returned error can't find the container with id 00a59afa14b6063d2c4ea9aa98c55c3e7be7d3e5a8f6d68e18eba056765100e2
Oct 30 22:04:09.150829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.150796    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:09.150928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.150871    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:09.150928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.150902    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:09.288739 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.288659    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp" podUID="93885823-090d-4eab-9518-98282531c551" containerName="payload-processing" containerID="cri-o://28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08" gracePeriod=25
Oct 30 22:04:09.584869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.584829    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:04:09.718275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.718231    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g\") pod \"93885823-090d-4eab-9518-98282531c551\" (UID: \"93885823-090d-4eab-9518-98282531c551\") "
Oct 30 22:04:09.718481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.718307    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials\") pod \"93885823-090d-4eab-9518-98282531c551\" (UID: \"93885823-090d-4eab-9518-98282531c551\") "
Oct 30 22:04:09.718546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.718530    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume\") pod \"93885823-090d-4eab-9518-98282531c551\" (UID: \"93885823-090d-4eab-9518-98282531c551\") "
Oct 30 22:04:09.718619 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.718576    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume\") pod \"93885823-090d-4eab-9518-98282531c551\" (UID: \"93885823-090d-4eab-9518-98282531c551\") "
Oct 30 22:04:09.718946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.718902    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume" pod "93885823-090d-4eab-9518-98282531c551" (UID: "93885823-090d-4eab-9518-98282531c551"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:09.719053 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.719021    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume" pod "93885823-090d-4eab-9518-98282531c551" (UID: "93885823-090d-4eab-9518-98282531c551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:09.719443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.719417    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:09.719543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.719448    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/93885823-090d-4eab-9518-98282531c551-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:09.720586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.720559    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g" pod "93885823-090d-4eab-9518-98282531c551" (UID: "93885823-090d-4eab-9518-98282531c551"). InnerVolumeSpecName "kube-api-access-89w6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.720785 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.720764    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials" pod "93885823-090d-4eab-9518-98282531c551" (UID: "93885823-090d-4eab-9518-98282531c551"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.815163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.813805    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:04:09.821712 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.821471    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89w6g\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-kube-api-access-89w6g\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:09.821712 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.821520    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/93885823-090d-4eab-9518-98282531c551-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:09.915654 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.915588    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" event={"ID":"f4fe97be-6faf-4aad-a571-6307e44e00e4","Type":"ContainerStarted","Data":"b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00"}
Oct 30 22:04:09.915884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.915685    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" event={"ID":"f4fe97be-6faf-4aad-a571-6307e44e00e4","Type":"ContainerStarted","Data":"00a59afa14b6063d2c4ea9aa98c55c3e7be7d3e5a8f6d68e18eba056765100e2"}
Oct 30 22:04:09.918165 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.918135    6695 generic.go:356] "Generic (PLEG): container finished" podID="93885823-090d-4eab-9518-98282531c551" containerID="28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08" exitCode=0
Oct 30 22:04:09.918335 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.918194    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"
Oct 30 22:04:09.918335 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.918215    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp" event={"ID":"93885823-090d-4eab-9518-98282531c551","Type":"ContainerDied","Data":"28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08"}
Oct 30 22:04:09.918335 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.918282    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp" event={"ID":"93885823-090d-4eab-9518-98282531c551","Type":"ContainerDied","Data":"d2f7b245696a45673217d79df438ed2bffa6bf93d69843782093ef4f3ea2973a"}
Oct 30 22:04:09.918335 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.918307    6695 scope.go:122] "RemoveContainer" containerID="28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08"
Oct 30 22:04:09.931892 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.931865    6695 scope.go:122] "RemoveContainer" containerID="28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08"
Oct 30 22:04:09.932300 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:04:09.932275    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08\": container with ID starting with 28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08 not found: ID does not exist" containerID="28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08"
Oct 30 22:04:09.932393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.932311    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08"} err="failed to get container status \"28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08\": rpc error: code = NotFound desc = could not find container \"28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08\": container with ID starting with 28f950e73beb63edf1ffc33b0a53df05a145e1cf8302387e22cd9b6d61643f08 not found: ID does not exist"
Oct 30 22:04:09.956904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:09.956836    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podStartSLOduration=1.9568223169999999 podStartE2EDuration="1.956822317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:09.955910293 +0000 UTC m=+3232.878368601" watchObservedRunningTime="2026-10-30 22:04:09.956822317 +0000 UTC m=+3232.879280614"
Oct 30 22:04:10.014850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.014746    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"]
Oct 30 22:04:10.022234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.022192    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-427a6e/payload-processing-external-model-e2e-extmod-a-427a6e-698dmd2bp"]
Oct 30 22:04:10.489565 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.489525    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"]
Oct 30 22:04:10.490454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.490433    6695 cpu_manager.go:414] "containerMap: removing container" podUID="93885823-090d-4eab-9518-98282531c551" containerName="payload-processing"
Oct 30 22:04:10.490514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.490457    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93885823-090d-4eab-9518-98282531c551" containerName="payload-processing"
Oct 30 22:04:10.490628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.490617    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="93885823-090d-4eab-9518-98282531c551" containerName="payload-processing"
Oct 30 22:04:10.495668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.495641    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.509998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.509967    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:10.510148 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.510071    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:04:10.528004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.526866    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:10.528004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.526958    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:10.546152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.546118    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"]
Oct 30 22:04:10.647981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.647938    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.647981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.647987    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdspp\" (UniqueName: \"kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.648445 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.648069    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.749838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.749737    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.749838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.749790    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdspp\" (UniqueName: \"kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.749838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.749810    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.750749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.750724    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.752357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.752329    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.769849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.769807    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdspp\" (UniqueName: \"kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp\") pod \"maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.833628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.825035    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:10.835627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.835420    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:10.837464 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.837433    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body=
Oct 30 22:04:10.837677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:10.837508    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused"
Oct 30 22:04:11.148500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.148470    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"]
Oct 30 22:04:11.152696 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:04:11.152657    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce909d1_0040_4dfd_9952_e6504a5ecbd0.slice/crio-b17743e699d0f8dd37b76ebcaa5bc8063aa6c79d25da7515fdc20e4cac3c2abc WatchSource:0}: Error finding container b17743e699d0f8dd37b76ebcaa5bc8063aa6c79d25da7515fdc20e4cac3c2abc: Status 404 returned error can't find the container with id b17743e699d0f8dd37b76ebcaa5bc8063aa6c79d25da7515fdc20e4cac3c2abc
Oct 30 22:04:11.247648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.247583    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93885823-090d-4eab-9518-98282531c551" path="/var/lib/kubelet/pods/93885823-090d-4eab-9518-98282531c551/volumes"
Oct 30 22:04:11.835629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.832447    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body=
Oct 30 22:04:11.835629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.832508    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused"
Oct 30 22:04:11.838670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.838043    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:04:11.964525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.964435    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8" event={"ID":"cce909d1-0040-4dfd-9952-e6504a5ecbd0","Type":"ContainerStarted","Data":"72311d1e86ab46765b1e66f2934be8c2e2acbc8918e8e7586c517ea2980414d1"}
Oct 30 22:04:11.964525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:11.964491    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8" event={"ID":"cce909d1-0040-4dfd-9952-e6504a5ecbd0","Type":"ContainerStarted","Data":"b17743e699d0f8dd37b76ebcaa5bc8063aa6c79d25da7515fdc20e4cac3c2abc"}
Oct 30 22:04:12.036319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:12.036246    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8" podStartSLOduration=2.03622326 podStartE2EDuration="2.03622326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:12.010402828 +0000 UTC m=+3234.932861127" watchObservedRunningTime="2026-10-30 22:04:12.03622326 +0000 UTC m=+3234.958681559"
Oct 30 22:04:12.817629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:12.815080    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body=
Oct 30 22:04:12.817629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:12.815148    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused"
Oct 30 22:04:12.974651 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:12.974588    6695 generic.go:356] "Generic (PLEG): container finished" podID="cce909d1-0040-4dfd-9952-e6504a5ecbd0" containerID="72311d1e86ab46765b1e66f2934be8c2e2acbc8918e8e7586c517ea2980414d1" exitCode=0
Oct 30 22:04:12.975051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:12.974663    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8" event={"ID":"cce909d1-0040-4dfd-9952-e6504a5ecbd0","Type":"ContainerDied","Data":"72311d1e86ab46765b1e66f2934be8c2e2acbc8918e8e7586c517ea2980414d1"}
Oct 30 22:04:13.818630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:13.818034    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.818630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:13.818100    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused"
Oct 30 22:04:14.195253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.195216    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:14.208183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.208136    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp\" (UniqueName: \"kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp\") pod \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") "
Oct 30 22:04:14.208354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.208340    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca\") pod \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") "
Oct 30 22:04:14.208394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.208379    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca\") pod \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\" (UID: \"cce909d1-0040-4dfd-9952-e6504a5ecbd0\") "
Oct 30 22:04:14.208788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.208755    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca" pod "cce909d1-0040-4dfd-9952-e6504a5ecbd0" (UID: "cce909d1-0040-4dfd-9952-e6504a5ecbd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:14.210378 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.210347    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp" pod "cce909d1-0040-4dfd-9952-e6504a5ecbd0" (UID: "cce909d1-0040-4dfd-9952-e6504a5ecbd0"). InnerVolumeSpecName "kube-api-access-sdspp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:14.210501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.210429    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca" pod "cce909d1-0040-4dfd-9952-e6504a5ecbd0" (UID: "cce909d1-0040-4dfd-9952-e6504a5ecbd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:14.324689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.310882    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:14.324689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.310929    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce909d1-0040-4dfd-9952-e6504a5ecbd0-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:14.324689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.310944    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdspp\" (UniqueName: \"kubernetes.io/projected/cce909d1-0040-4dfd-9952-e6504a5ecbd0-kube-api-access-sdspp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:14.831633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.823083    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.831633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.823153    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused"
Oct 30 22:04:14.994090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.994054    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8" event={"ID":"cce909d1-0040-4dfd-9952-e6504a5ecbd0","Type":"ContainerDied","Data":"b17743e699d0f8dd37b76ebcaa5bc8063aa6c79d25da7515fdc20e4cac3c2abc"}
Oct 30 22:04:14.994283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.994132    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17743e699d0f8dd37b76ebcaa5bc8063aa6c79d25da7515fdc20e4cac3c2abc"
Oct 30 22:04:14.994283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:14.994066    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"
Oct 30 22:04:15.818361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:15.818323    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.818908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:15.818384    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused"
Oct 30 22:04:16.729485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.729444    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"]
Oct 30 22:04:16.730419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.730393    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cce909d1-0040-4dfd-9952-e6504a5ecbd0" containerName="revoke-keys"
Oct 30 22:04:16.730419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.730419    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce909d1-0040-4dfd-9952-e6504a5ecbd0" containerName="revoke-keys"
Oct 30 22:04:16.730696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.730680    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce909d1-0040-4dfd-9952-e6504a5ecbd0" containerName="revoke-keys"
Oct 30 22:04:16.735408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.735383    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.755451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.755417    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"]
Oct 30 22:04:16.760636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.760597    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-50ab77-openshift-default-dockercfg-tt79b\""
Oct 30 22:04:16.769465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769416    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6kd\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769481    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769572    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769648    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769679    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769711    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769789    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769841    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.769936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.769884    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.816956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.816923    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:16.817416 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.817394    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:16.818269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.818251    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:04:16.871258 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871218    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.871814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871314    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.871814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871353    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.871814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871379    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.871814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871411    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.871814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871469    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.871814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871515    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871572    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871918    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.871815    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.872127    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.872186    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.872256    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.872721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.872497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6kd\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.874117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.874095    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.874437 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.874416    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.894362 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.894323    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:16.894822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:16.894803    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6kd\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:17.057377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.057342    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:17.403801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.403766    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"]
Oct 30 22:04:17.406132 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:04:17.406103    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5675ebbf_042a_4a91_ab8c_1adacaf45777.slice/crio-e324469d58df23f99908a4fa9afa2451276fe41be927941ee6b13216229c7178 WatchSource:0}: Error finding container e324469d58df23f99908a4fa9afa2451276fe41be927941ee6b13216229c7178: Status 404 returned error can't find the container with id e324469d58df23f99908a4fa9afa2451276fe41be927941ee6b13216229c7178
Oct 30 22:04:17.409393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.409356    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:17.409496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.409443    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:17.409496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.409473    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:17.439597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.439560    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"]
Oct 30 22:04:17.446792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.446767    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.462499 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.462465    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b31518-openshift-default-dockercfg-xdgb8\""
Oct 30 22:04:17.481427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.481390    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.481642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.481533    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.481642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.481580    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztcr\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.481805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.481702    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.481805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.481796    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.481993 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.481973    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.482087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.482002    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.482087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.482071    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.482224 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.482118    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.489923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.489885    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"]
Oct 30 22:04:17.584658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584480    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584560    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584594    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztcr\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584664    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584751    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584896    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584924    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.584983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.584967    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.585267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.585017    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.585453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.585429    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.586130 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.586099    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.586407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.586390    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.586664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.586640    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.586780 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.586614    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.588873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.588841    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.590820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.590797    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.615001 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.614971    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.616361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.616330    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztcr\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:17.791109 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.791070    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b31518-openshift-default-dockercfg-xdgb8\""
Oct 30 22:04:17.798173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:17.798133    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:18.079028 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.078910    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" event={"ID":"5675ebbf-042a-4a91-ab8c-1adacaf45777","Type":"ContainerStarted","Data":"c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934"}
Oct 30 22:04:18.079028 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.078987    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" event={"ID":"5675ebbf-042a-4a91-ab8c-1adacaf45777","Type":"ContainerStarted","Data":"e324469d58df23f99908a4fa9afa2451276fe41be927941ee6b13216229c7178"}
Oct 30 22:04:18.124201 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.124141    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podStartSLOduration=2.124121759 podStartE2EDuration="2.124121759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:18.119834744 +0000 UTC m=+3241.042293078" watchObservedRunningTime="2026-10-30 22:04:18.124121759 +0000 UTC m=+3241.046580108"
Oct 30 22:04:18.136293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.136257    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"]
Oct 30 22:04:18.137220 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:04:18.137189    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7cc97d_228d_4f75_a3b9_914cf91dac07.slice/crio-ea2b5513475e188a4477deab518637e138322163833bfa66113945a23be444f1 WatchSource:0}: Error finding container ea2b5513475e188a4477deab518637e138322163833bfa66113945a23be444f1: Status 404 returned error can't find the container with id ea2b5513475e188a4477deab518637e138322163833bfa66113945a23be444f1
Oct 30 22:04:18.140299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.140265    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:18.140390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.140335    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:18.140390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.140366    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:18.547660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.547092    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"]
Oct 30 22:04:18.560528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.557065    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"]
Oct 30 22:04:18.560528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.557238    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.578618 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.577992    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ed0866-openshift-default-dockercfg-sl7dd\""
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620394    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620461    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620541    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbm4t\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620591    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620638    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620704    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620781    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620870    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.634628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.620900    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.723701 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.723656    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbm4t\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.724365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.724332    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.724580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.724560    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.725729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.725376    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.725729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.725473    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.725729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.725561    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.725729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.725588    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.726008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.725843    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.726008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.725885    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.729343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.726861    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.729343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.727503    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.729343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.728334    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.729343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.728567    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.729343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.729191    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.743169 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.743132    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.743781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.743755    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.747304 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.747278    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbm4t\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.747422 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.747306    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:18.886126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:18.886022    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:19.069128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.067270    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:19.071771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.070225    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.071771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.070305    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:19.091782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.091693    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" event={"ID":"ee7cc97d-228d-4f75-a3b9-914cf91dac07","Type":"ContainerStarted","Data":"2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d"}
Oct 30 22:04:19.091782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.091771    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" event={"ID":"ee7cc97d-228d-4f75-a3b9-914cf91dac07","Type":"ContainerStarted","Data":"ea2b5513475e188a4477deab518637e138322163833bfa66113945a23be444f1"}
Oct 30 22:04:19.295041 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.294983    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podStartSLOduration=2.294964997 podStartE2EDuration="2.294964997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:19.155841913 +0000 UTC m=+3242.078300212" watchObservedRunningTime="2026-10-30 22:04:19.294964997 +0000 UTC m=+3242.217423303"
Oct 30 22:04:19.297031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.296912    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"]
Oct 30 22:04:19.299431 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:04:19.299394    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806f2d5a_e7ee_48a8_b90d_77b761d67260.slice/crio-90132d21a842dc8068406c467fcb0fa7b37701b4142fa997c951e4ad1303652e WatchSource:0}: Error finding container 90132d21a842dc8068406c467fcb0fa7b37701b4142fa997c951e4ad1303652e: Status 404 returned error can't find the container with id 90132d21a842dc8068406c467fcb0fa7b37701b4142fa997c951e4ad1303652e
Oct 30 22:04:19.302884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.302845    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:19.302979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.302936    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:19.303024 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.302980    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:19.799223 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.799135    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:19.800294 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.800273    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.800475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.800458    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:19.881244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.881157    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"]
Oct 30 22:04:19.886367 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.886340    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"]
Oct 30 22:04:19.886718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.886704    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.902081 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.902056    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-49738b-openshift-default-dockercfg-h4mdb\""
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982496    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982578    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982638    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982721    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982839    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982885    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.982972    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.983150    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49v4q\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:19.983907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:19.983183    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.071824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.066788    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.071824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.066860    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:20.085793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085643    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49v4q\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.085793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085684    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.085793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085717    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.085793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085751    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.085793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085775    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085830    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085926    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.085998    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.086413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.086359    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.087350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.087323    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.088003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.087950    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.088423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.088398    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.088739 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.088716    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.091520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.091417    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.091989 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.091811    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.102152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.102101    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" event={"ID":"806f2d5a-e7ee-48a8-b90d-77b761d67260","Type":"ContainerStarted","Data":"ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed"}
Oct 30 22:04:20.102328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.102169    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" event={"ID":"806f2d5a-e7ee-48a8-b90d-77b761d67260","Type":"ContainerStarted","Data":"90132d21a842dc8068406c467fcb0fa7b37701b4142fa997c951e4ad1303652e"}
Oct 30 22:04:20.107347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.107314    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.108223 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.108188    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49v4q\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.162357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.160119    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podStartSLOduration=2.160098147 podStartE2EDuration="2.160098147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:20.159074498 +0000 UTC m=+3243.081532809" watchObservedRunningTime="2026-10-30 22:04:20.160098147 +0000 UTC m=+3243.082556446"
Oct 30 22:04:20.268626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.268303    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:20.886638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.830016    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.886638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.830153    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:20.965690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:20.961903    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:21.005141 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.005083    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.005368 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.005147    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:21.043003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.012300    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:21.043003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.012399    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:21.043003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.012442    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:21.091630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.069025    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.092077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.092034    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:21.092564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.090144    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"]
Oct 30 22:04:21.763629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.761188    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"]
Oct 30 22:04:21.763629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.761269    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" event={"ID":"97f58aad-5421-4be1-ab2b-d26d0ceb46e5","Type":"ContainerStarted","Data":"032b5bf6b5c5b7333be0f510a99487eda7b6b7c45cdbce2f4a27c7cdfa2bead0"}
Oct 30 22:04:21.763629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.761303    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"]
Oct 30 22:04:21.803746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.803005    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.803746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.803064    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:21.890681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.890384    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.890681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:21.890447    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:22.084626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.082388    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.084626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.082447    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:22.795243 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.795202    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" event={"ID":"97f58aad-5421-4be1-ab2b-d26d0ceb46e5","Type":"ContainerStarted","Data":"c2cfe4c2625709ca3053e5769f05bb7f750e147ba834e6d88bdf6032ef16d8dc"}
Oct 30 22:04:22.800357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.800247    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.800741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.800714    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:22.823763 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.823709    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 22:04:22.864756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.864709    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:22.867863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.867801    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podStartSLOduration=3.867780877 podStartE2EDuration="3.867780877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:22.852100183 +0000 UTC m=+3245.774558484" watchObservedRunningTime="2026-10-30 22:04:22.867780877 +0000 UTC m=+3245.790239176"
Oct 30 22:04:22.895572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.895162    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.896256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:22.896221    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:23.075627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.075511    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.079689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.075854    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:23.214852 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.214688    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:23.276624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.273986    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:23.276624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.274130    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.276624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.274177    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:23.367689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.366850    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:23.457628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.456722    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:23.600730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.600440    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:23.642683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.641779    6695 scope.go:122] "RemoveContainer" containerID="a9a00650ff61d36a9f804a53866c347915f02329a12adc325d6a952e3f476720"
Oct 30 22:04:23.826726 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.826512    6695 scope.go:122] "RemoveContainer" containerID="f6c85f791feaedc4905c32a00834bb4fdc645878223b60aa378c989e07fbfc55"
Oct 30 22:04:23.861750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.859714    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:23.861750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.860565    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 22:04:23.870899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.870148    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.870899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.870208    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:23.892656 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.887021    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.893039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:23.892958    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:24.043413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.043370    6695 scope.go:122] "RemoveContainer" containerID="a47ca6f4a1d4edf835e2e54df5768c2b31c23fc6085cf98ee20f91c660749082"
Oct 30 22:04:24.115476 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.115432    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:24.117541 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.117504    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:24.552672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.531030    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:04:24.552672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.543223    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:24.552672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.543306    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:24.639568 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.613051    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:24.639568 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.638068    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 22:04:24.723628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.721966    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:24.907663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.899713    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 22:04:24.907663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.905065    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:24.907663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:24.905140    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:25.096685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.079887    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.096685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.079996    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:25.154685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.129712    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.541640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.536524    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.541945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.540291    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.569958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.569902    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.634671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.634623    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.635873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.635837    6695 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.723080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.723011    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.803686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.787946    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.803686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.788181    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:25.824405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.824358    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:25.884023 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.883987    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.884285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.884248    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:25.902091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.902032    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:25.903596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.903552    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.903897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:25.903871    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:26.021871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.016755    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 22:04:26.021871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.016823    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 22:04:26.061686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.022495    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:26.093150 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.069790    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:26.093150 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.069840    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:26.375984 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.375937    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:26.376216 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.376012    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:26.598152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.598049    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:26.598886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.598861    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 22:04:26.694644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.668349    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:26.794630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.768685    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerName="payload-processing" containerID="cri-o://3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c" gracePeriod=25
Oct 30 22:04:26.794630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.768941    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerName="payload-pre-processing" containerID="cri-o://75767dca364cec633c0591e2f9439912e0de6563a39f213217dd504671be4d65" gracePeriod=25
Oct 30 22:04:26.833670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.804041    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:26.833670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.804111    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:26.833670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.832711    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 30 22:04:26.885693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.730854    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:26.954627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.934479    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:26.955867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.955004    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-rc9jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:26.955867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.955070    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.007670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:26.994430    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.008982 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.008527    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:27.080783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.079023    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.080783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.079770    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 22:04:27.080783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.080195    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.080783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.080240    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:27.214688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.195725    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fj2hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:27.214688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.195796    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.467687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.459157    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.467687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.459221    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:27.598776 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.598732    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.681499 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.680419    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:04:27.796634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.796452    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 22:04:27.805640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.803033    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.805640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.803098    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:27.838630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.834868    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.838630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.837264    6695 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-j6jq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:27.838630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.837321    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.869821 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.869721    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:27.984631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.958869    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.985391 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.985360    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:27.985592 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:27.985047    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:28.076994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.076949    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused" start-of-body=
Oct 30 22:04:28.080752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.080693    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": dial tcp 10.134.1.180:15021: connect: connection refused"
Oct 30 22:04:28.300066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.300024    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:28.302114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.302073    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:28.607348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.607211    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:28.780198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.780155    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:28.789319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.789258    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:28.807723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.807683    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:28.808051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.808019    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:28.839642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.838718    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:28.880733 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.877917    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:28.893820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.893777    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:28.894120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:28.894087    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:29.024630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.006488    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 22:04:29.286644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.275953    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:29.286644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.276018    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:29.623465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.623360    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:29.789636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.788801    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:29.793633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.790891    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:29.793633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.791351    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 22:04:29.803626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.802256    6695 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused" start-of-body=
Oct 30 22:04:29.803626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.802320    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.181:15021/healthz/ready\": dial tcp 10.134.1.181:15021: connect: connection refused"
Oct 30 22:04:29.862694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.860723    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:29.882411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:29.881773    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:04:30.086675 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.063689    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 30 22:04:30.148352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.148251    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.149990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.148597    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:30.229640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.229489    6695 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:30.229640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.229562    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:30.252019 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.251979    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:30.280732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.272754    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.280732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.272827    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:30.280732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.274351    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:30.289009 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.288968    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:04:30.640597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.640535    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:30.708828 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.708787    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 22:04:30.709186 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.709163    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 22:04:30.769018 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.768975    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 22:04:30.781078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.781046    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 22:04:30.799871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.798411    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 22:04:30.883628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.882337    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:30.905832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.905710    6695 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.906149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:30.906095    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.182:15021/healthz/ready\": dial tcp 10.134.1.182:15021: connect: connection refused"
Oct 30 22:04:31.005764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.005731    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 22:04:31.067433 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.067391    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:31.070383 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.069170    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:31.073172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.072175    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:04:31.277821 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.277773    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:31.278047 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.277850    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:31.280811 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:04:31.280768    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:04:31.892753 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.892722    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:31.893304 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.893288    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:31.894246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:31.894221    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:04:32.273301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:32.273267    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:32.273639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:32.273578    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:33.270127 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:33.270080    6695 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused" start-of-body=
Oct 30 22:04:33.270650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:33.270154    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.183:15021/healthz/ready\": dial tcp 10.134.1.183:15021: connect: connection refused"
Oct 30 22:04:34.005686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:34.005632    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:04:34.006039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:34.006013    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:04:34.649403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:34.649368    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:34.649983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:34.649966    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:34.651512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:34.651492    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:04:44.028627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:44.028299    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:04:44.066652 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:44.066583    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:04:46.880200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:46.880157    6695 generic.go:356] "Generic (PLEG): container finished" podID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerID="75767dca364cec633c0591e2f9439912e0de6563a39f213217dd504671be4d65" exitCode=0
Oct 30 22:04:46.880650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:46.880221    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" event={"ID":"c2ce44aa-6dc9-47a3-98d2-5e55f569416a","Type":"ContainerDied","Data":"75767dca364cec633c0591e2f9439912e0de6563a39f213217dd504671be4d65"}
Oct 30 22:04:47.147089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.147047    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 22:04:47.230503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.230437    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume\") pod \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") "
Oct 30 22:04:47.231247 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.231204    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx\" (UniqueName: \"kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx\") pod \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\" (UID: \"c2ce44aa-6dc9-47a3-98d2-5e55f569416a\") "
Oct 30 22:04:47.232891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.231087    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume" pod "c2ce44aa-6dc9-47a3-98d2-5e55f569416a" (UID: "c2ce44aa-6dc9-47a3-98d2-5e55f569416a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:47.240381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.240345    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 22:04:47.240750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.240718    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx" pod "c2ce44aa-6dc9-47a3-98d2-5e55f569416a" (UID: "c2ce44aa-6dc9-47a3-98d2-5e55f569416a"). InnerVolumeSpecName "kube-api-access-v9hlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:47.333868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.333754    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume\") pod \"f160d273-95ba-4579-a0f5-b463d27fdaf2\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") "
Oct 30 22:04:47.334080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.334057    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw\" (UniqueName: \"kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw\") pod \"f160d273-95ba-4579-a0f5-b463d27fdaf2\" (UID: \"f160d273-95ba-4579-a0f5-b463d27fdaf2\") "
Oct 30 22:04:47.334295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.334249    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume" pod "f160d273-95ba-4579-a0f5-b463d27fdaf2" (UID: "f160d273-95ba-4579-a0f5-b463d27fdaf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:47.335323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.335294    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f160d273-95ba-4579-a0f5-b463d27fdaf2-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:47.335450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.335329    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:47.335450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.335344    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hlx\" (UniqueName: \"kubernetes.io/projected/c2ce44aa-6dc9-47a3-98d2-5e55f569416a-kube-api-access-v9hlx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:47.337077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.337048    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw" pod "f160d273-95ba-4579-a0f5-b463d27fdaf2" (UID: "f160d273-95ba-4579-a0f5-b463d27fdaf2"). InnerVolumeSpecName "kube-api-access-6grxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:47.437087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.437039    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grxw\" (UniqueName: \"kubernetes.io/projected/f160d273-95ba-4579-a0f5-b463d27fdaf2-kube-api-access-6grxw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:04:47.890830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.890792    6695 generic.go:356] "Generic (PLEG): container finished" podID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerID="3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c" exitCode=0
Oct 30 22:04:47.891326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.890841    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"
Oct 30 22:04:47.891326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.890878    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" event={"ID":"f160d273-95ba-4579-a0f5-b463d27fdaf2","Type":"ContainerDied","Data":"3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c"}
Oct 30 22:04:47.891326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.890932    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z" event={"ID":"f160d273-95ba-4579-a0f5-b463d27fdaf2","Type":"ContainerDied","Data":"157f0501509a8ec0e693c6e47894d36e7d41440b7dbeaf0b1d74360d48e35ce1"}
Oct 30 22:04:47.891326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.890961    6695 scope.go:122] "RemoveContainer" containerID="3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c"
Oct 30 22:04:47.892684 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.892656    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq" event={"ID":"c2ce44aa-6dc9-47a3-98d2-5e55f569416a","Type":"ContainerDied","Data":"9aae2650b5e943710b3b13253567fb8eb3cc47df746ce7c5beefe5168f835fb1"}
Oct 30 22:04:47.892798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.892673    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"
Oct 30 22:04:47.903117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.903092    6695 scope.go:122] "RemoveContainer" containerID="3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c"
Oct 30 22:04:47.903443 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:04:47.903413    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c\": container with ID starting with 3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c not found: ID does not exist" containerID="3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c"
Oct 30 22:04:47.903546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.903443    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c"} err="failed to get container status \"3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c\": rpc error: code = NotFound desc = could not find container \"3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c\": container with ID starting with 3785b0dd72576c4e39e09b9eb838076e90ed2bfa081208d3c6e812de7347151c not found: ID does not exist"
Oct 30 22:04:47.903546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.903463    6695 scope.go:122] "RemoveContainer" containerID="75767dca364cec633c0591e2f9439912e0de6563a39f213217dd504671be4d65"
Oct 30 22:04:47.944747 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.944710    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"]
Oct 30 22:04:47.950673 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.950630    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-427a6e-84b5f948bc-cprtq"]
Oct 30 22:04:47.979829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.979793    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"]
Oct 30 22:04:47.984426 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:47.984385    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-427a6e-598d6b577-m7x9z"]
Oct 30 22:04:49.223337 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:49.223300    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" path="/var/lib/kubelet/pods/c2ce44aa-6dc9-47a3-98d2-5e55f569416a/volumes"
Oct 30 22:04:49.223872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:49.223855    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" path="/var/lib/kubelet/pods/f160d273-95ba-4579-a0f5-b463d27fdaf2/volumes"
Oct 30 22:04:52.839197 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:52.839153    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"]
Oct 30 22:04:52.851305 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:52.851269    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-427a6e-ps4r8"]
Oct 30 22:04:53.217365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:53.217270    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce909d1-0040-4dfd-9952-e6504a5ecbd0" path="/var/lib/kubelet/pods/cce909d1-0040-4dfd-9952-e6504a5ecbd0/volumes"
Oct 30 22:04:55.042345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:55.042290    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"]
Oct 30 22:04:55.052929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:55.052897    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74142843-n27s5"]
Oct 30 22:04:55.216796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:55.216759    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7ffce9-2236-4940-b5ee-dd4376ddf8cb" path="/var/lib/kubelet/pods/cb7ffce9-2236-4940-b5ee-dd4376ddf8cb/volumes"
Oct 30 22:04:56.445534 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.445494    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"]
Oct 30 22:04:56.446409 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.446392    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerName="payload-processing"
Oct 30 22:04:56.446462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.446410    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerName="payload-processing"
Oct 30 22:04:56.446462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.446446    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerName="payload-pre-processing"
Oct 30 22:04:56.446462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.446452    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerName="payload-pre-processing"
Oct 30 22:04:56.446635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.446598    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f160d273-95ba-4579-a0f5-b463d27fdaf2" containerName="payload-processing"
Oct 30 22:04:56.446635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.446633    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ce44aa-6dc9-47a3-98d2-5e55f569416a" containerName="payload-pre-processing"
Oct 30 22:04:56.451702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.451683    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.470893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.470856    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"]
Oct 30 22:04:56.476362 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.476331    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:56.476362 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.476359    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:04:56.476573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.476376    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:04:56.477371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.477351    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:04:56.561764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.561732    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.562267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.562233    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.562429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.562405    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5fv\" (UniqueName: \"kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.664405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.664370    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.664655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.664457    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5fv\" (UniqueName: \"kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.664655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.664505    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.665225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.665195    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.667002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.666972    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.683316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.683280    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5fv\" (UniqueName: \"kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv\") pod \"maas-api-revoke-sub-c4fb2eff-t6k8r\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:56.767410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:56.767327    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:04:57.081196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:57.081163    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"]
Oct 30 22:04:57.083239 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:04:57.083213    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7556234a_e5de_4646_85c2_d97661cfa804.slice/crio-56c15a1129b05cde258e7cfc7bd9328eee788a75b60d05ddd7bf63e664f7a492 WatchSource:0}: Error finding container 56c15a1129b05cde258e7cfc7bd9328eee788a75b60d05ddd7bf63e664f7a492: Status 404 returned error can't find the container with id 56c15a1129b05cde258e7cfc7bd9328eee788a75b60d05ddd7bf63e664f7a492
Oct 30 22:04:57.990902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:57.990853    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r" event={"ID":"7556234a-e5de-4646-85c2-d97661cfa804","Type":"ContainerStarted","Data":"5ff115e20404b3183bdd23c5e5f77295a741f0949335d4f034eb3178cbf92ce6"}
Oct 30 22:04:57.991370 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:57.990930    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r" event={"ID":"7556234a-e5de-4646-85c2-d97661cfa804","Type":"ContainerStarted","Data":"56c15a1129b05cde258e7cfc7bd9328eee788a75b60d05ddd7bf63e664f7a492"}
Oct 30 22:04:59.001758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:59.001722    6695 generic.go:356] "Generic (PLEG): container finished" podID="7556234a-e5de-4646-85c2-d97661cfa804" containerID="5ff115e20404b3183bdd23c5e5f77295a741f0949335d4f034eb3178cbf92ce6" exitCode=0
Oct 30 22:04:59.002181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:59.001800    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r" event={"ID":"7556234a-e5de-4646-85c2-d97661cfa804","Type":"ContainerDied","Data":"5ff115e20404b3183bdd23c5e5f77295a741f0949335d4f034eb3178cbf92ce6"}
Oct 30 22:04:59.664080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:59.664041    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"]
Oct 30 22:04:59.664777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:04:59.664305    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" containerID="cri-o://b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00" gracePeriod=30
Oct 30 22:05:00.199592 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.199553    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:05:00.269908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.269872    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"]
Oct 30 22:05:00.270863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.270841    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7556234a-e5de-4646-85c2-d97661cfa804" containerName="revoke-keys"
Oct 30 22:05:00.270863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.270862    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7556234a-e5de-4646-85c2-d97661cfa804" containerName="revoke-keys"
Oct 30 22:05:00.271051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.271005    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7556234a-e5de-4646-85c2-d97661cfa804" containerName="revoke-keys"
Oct 30 22:05:00.275017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.274995    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.297686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.297586    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"]
Oct 30 22:05:00.345137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.345095    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca\") pod \"7556234a-e5de-4646-85c2-d97661cfa804\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") "
Oct 30 22:05:00.345349 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.345176    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca\") pod \"7556234a-e5de-4646-85c2-d97661cfa804\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") "
Oct 30 22:05:00.345460 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.345435    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv\" (UniqueName: \"kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv\") pod \"7556234a-e5de-4646-85c2-d97661cfa804\" (UID: \"7556234a-e5de-4646-85c2-d97661cfa804\") "
Oct 30 22:05:00.345564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.345542    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca" pod "7556234a-e5de-4646-85c2-d97661cfa804" (UID: "7556234a-e5de-4646-85c2-d97661cfa804"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:00.346940 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.346660    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:00.348198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.348164    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv" pod "7556234a-e5de-4646-85c2-d97661cfa804" (UID: "7556234a-e5de-4646-85c2-d97661cfa804"). InnerVolumeSpecName "kube-api-access-wb5fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:00.352801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.352767    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca" pod "7556234a-e5de-4646-85c2-d97661cfa804" (UID: "7556234a-e5de-4646-85c2-d97661cfa804"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:00.448468 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.448423    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.448692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.448664    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.448747 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.448719    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljj2f\" (UniqueName: \"kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.448918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.448899    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5fv\" (UniqueName: \"kubernetes.io/projected/7556234a-e5de-4646-85c2-d97661cfa804-kube-api-access-wb5fv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:00.448992 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.448924    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7556234a-e5de-4646-85c2-d97661cfa804-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:00.550741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.550641    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.550741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.550698    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljj2f\" (UniqueName: \"kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.550996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.550839    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.551394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.551368    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.553272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.553248    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.570351 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.570321    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljj2f\" (UniqueName: \"kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f\") pod \"maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.600682 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.600636    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:00.916992 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:00.916959    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"]
Oct 30 22:05:00.918102 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:00.918079    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8831486b_dd73_4155_91eb_8df9e07541d8.slice/crio-27ed77a2fbc21d9038044039852da12fe7a85d4e9f64eb49d62b289c5479ae4c WatchSource:0}: Error finding container 27ed77a2fbc21d9038044039852da12fe7a85d4e9f64eb49d62b289c5479ae4c: Status 404 returned error can't find the container with id 27ed77a2fbc21d9038044039852da12fe7a85d4e9f64eb49d62b289c5479ae4c
Oct 30 22:05:01.022125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:01.022078    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh" event={"ID":"8831486b-dd73-4155-91eb-8df9e07541d8","Type":"ContainerStarted","Data":"27ed77a2fbc21d9038044039852da12fe7a85d4e9f64eb49d62b289c5479ae4c"}
Oct 30 22:05:01.023510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:01.023489    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"
Oct 30 22:05:01.023644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:01.023527    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r" event={"ID":"7556234a-e5de-4646-85c2-d97661cfa804","Type":"ContainerDied","Data":"56c15a1129b05cde258e7cfc7bd9328eee788a75b60d05ddd7bf63e664f7a492"}
Oct 30 22:05:01.023644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:01.023570    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c15a1129b05cde258e7cfc7bd9328eee788a75b60d05ddd7bf63e664f7a492"
Oct 30 22:05:01.136871 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:01.136821    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:05:01.819612 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:01.819560    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:05:01.820057 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:01.819666    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:05:02.033374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:02.033339    6695 generic.go:356] "Generic (PLEG): container finished" podID="8831486b-dd73-4155-91eb-8df9e07541d8" containerID="d2faaf1df73223a895aa1d35d3a710c409b8c7b258f7d0c5699512e8e5a70828" exitCode=0
Oct 30 22:05:02.033577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:02.033429    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh" event={"ID":"8831486b-dd73-4155-91eb-8df9e07541d8","Type":"ContainerDied","Data":"d2faaf1df73223a895aa1d35d3a710c409b8c7b258f7d0c5699512e8e5a70828"}
Oct 30 22:05:03.273266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.273226    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:03.402173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.402072    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca\") pod \"8831486b-dd73-4155-91eb-8df9e07541d8\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") "
Oct 30 22:05:03.402374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.402195    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca\") pod \"8831486b-dd73-4155-91eb-8df9e07541d8\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") "
Oct 30 22:05:03.402374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.402254    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f\" (UniqueName: \"kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f\") pod \"8831486b-dd73-4155-91eb-8df9e07541d8\" (UID: \"8831486b-dd73-4155-91eb-8df9e07541d8\") "
Oct 30 22:05:03.402552 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.402524    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca" pod "8831486b-dd73-4155-91eb-8df9e07541d8" (UID: "8831486b-dd73-4155-91eb-8df9e07541d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:03.403496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.403471    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:03.404881 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.404839    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca" pod "8831486b-dd73-4155-91eb-8df9e07541d8" (UID: "8831486b-dd73-4155-91eb-8df9e07541d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:03.404994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.404885    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f" pod "8831486b-dd73-4155-91eb-8df9e07541d8" (UID: "8831486b-dd73-4155-91eb-8df9e07541d8"). InnerVolumeSpecName "kube-api-access-ljj2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:03.505276 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.505237    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8831486b-dd73-4155-91eb-8df9e07541d8-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:03.505276 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:03.505274    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljj2f\" (UniqueName: \"kubernetes.io/projected/8831486b-dd73-4155-91eb-8df9e07541d8-kube-api-access-ljj2f\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:04.051582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:04.051550    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"
Oct 30 22:05:04.051810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:04.051590    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh" event={"ID":"8831486b-dd73-4155-91eb-8df9e07541d8","Type":"ContainerDied","Data":"27ed77a2fbc21d9038044039852da12fe7a85d4e9f64eb49d62b289c5479ae4c"}
Oct 30 22:05:04.051810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:04.051679    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ed77a2fbc21d9038044039852da12fe7a85d4e9f64eb49d62b289c5479ae4c"
Oct 30 22:05:05.014802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.014768    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:05:05.062208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.062172    6695 generic.go:356] "Generic (PLEG): container finished" podID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerID="b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00" exitCode=0
Oct 30 22:05:05.062424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.062224    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"
Oct 30 22:05:05.062424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.062256    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" event={"ID":"f4fe97be-6faf-4aad-a571-6307e44e00e4","Type":"ContainerDied","Data":"b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00"}
Oct 30 22:05:05.062424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.062330    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585" event={"ID":"f4fe97be-6faf-4aad-a571-6307e44e00e4","Type":"ContainerDied","Data":"00a59afa14b6063d2c4ea9aa98c55c3e7be7d3e5a8f6d68e18eba056765100e2"}
Oct 30 22:05:05.062424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.062362    6695 scope.go:122] "RemoveContainer" containerID="b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00"
Oct 30 22:05:05.076953 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.076928    6695 scope.go:122] "RemoveContainer" containerID="b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00"
Oct 30 22:05:05.077285 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:05.077263    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00\": container with ID starting with b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00 not found: ID does not exist" containerID="b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00"
Oct 30 22:05:05.077394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.077293    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00"} err="failed to get container status \"b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00\": rpc error: code = NotFound desc = could not find container \"b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00\": container with ID starting with b2eb685e7d84b6c9c9b065b88bb137b19642df01bbcce8a60c89b8376e427d00 not found: ID does not exist"
Oct 30 22:05:05.133819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.133776    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.133846    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.133924    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.133975    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134082    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134114    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:05.134315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134142    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134209    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134263    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6\") pod \"f4fe97be-6faf-4aad-a571-6307e44e00e4\" (UID: \"f4fe97be-6faf-4aad-a571-6307e44e00e4\") "
Oct 30 22:05:05.134718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134457    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:05.134873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134844    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:05.134955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134882    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:05.134955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.134941    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:05.135968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.135935    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.136102 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.135972    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fe97be-6faf-4aad-a571-6307e44e00e4-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.136102 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.135990    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.136102 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.136005    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.136102 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.136018    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.136845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.136812    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:05.136966 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.136942    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:05.137016 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.136976    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "kube-api-access-jvpx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:05.137176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.137151    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo" pod "f4fe97be-6faf-4aad-a571-6307e44e00e4" (UID: "f4fe97be-6faf-4aad-a571-6307e44e00e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:05.237572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.237525    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpx6\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-kube-api-access-jvpx6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.237572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.237571    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.237792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.237587    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.237792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.237615    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fe97be-6faf-4aad-a571-6307e44e00e4-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:05.406282 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.406214    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"]
Oct 30 22:05:05.411931 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:05.411894    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-427a6e-openshift-default-57556dbfb5-qf585"]
Oct 30 22:05:07.217400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.217353    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" path="/var/lib/kubelet/pods/f4fe97be-6faf-4aad-a571-6307e44e00e4/volumes"
Oct 30 22:05:07.313160 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.313108    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"]
Oct 30 22:05:07.314088 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.314063    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy"
Oct 30 22:05:07.314088 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.314088    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy"
Oct 30 22:05:07.314294 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.314141    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8831486b-dd73-4155-91eb-8df9e07541d8" containerName="revoke-keys"
Oct 30 22:05:07.314294 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.314149    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8831486b-dd73-4155-91eb-8df9e07541d8" containerName="revoke-keys"
Oct 30 22:05:07.314403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.314383    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fe97be-6faf-4aad-a571-6307e44e00e4" containerName="istio-proxy"
Oct 30 22:05:07.314464 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.314410    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8831486b-dd73-4155-91eb-8df9e07541d8" containerName="revoke-keys"
Oct 30 22:05:07.320452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.320421    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.335741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.335704    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:07.337082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.337050    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"]
Oct 30 22:05:07.341811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.341781    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:07.341811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.341800    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:07.346091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.346054    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:05:07.375215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.375176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.375215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.375213    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpmt\" (UniqueName: \"kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.375668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.375639    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.477963 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.477864    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.477963 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.477958    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.478222 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.477979    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpmt\" (UniqueName: \"kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.478702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.478678    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.480577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.480548    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.498170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.498109    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpmt\" (UniqueName: \"kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt\") pod \"maas-api-revoke-sub-fcfbb4f7-js4l7\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.641830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.641788    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:07.955615 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:07.955575    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"]
Oct 30 22:05:07.957873 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:07.957837    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005a4540_df13_41cb_8371_3820f879b8bf.slice/crio-c2b29cb984aa845bbcc0e229b3800c9d41a4d6eb43508627584104f351d1cf61 WatchSource:0}: Error finding container c2b29cb984aa845bbcc0e229b3800c9d41a4d6eb43508627584104f351d1cf61: Status 404 returned error can't find the container with id c2b29cb984aa845bbcc0e229b3800c9d41a4d6eb43508627584104f351d1cf61
Oct 30 22:05:08.093167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:08.093128    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7" event={"ID":"005a4540-df13-41cb-8371-3820f879b8bf","Type":"ContainerStarted","Data":"c2b29cb984aa845bbcc0e229b3800c9d41a4d6eb43508627584104f351d1cf61"}
Oct 30 22:05:09.067338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:09.067294    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"]
Oct 30 22:05:09.077635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:09.077572    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de3e169-x6cpr"]
Oct 30 22:05:09.103723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:09.103684    6695 generic.go:356] "Generic (PLEG): container finished" podID="005a4540-df13-41cb-8371-3820f879b8bf" containerID="ec967ca957f5c9b0f1e5b013a78fd3f921d9617a5de282e8056425bc73a16453" exitCode=0
Oct 30 22:05:09.103943 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:09.103774    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7" event={"ID":"005a4540-df13-41cb-8371-3820f879b8bf","Type":"ContainerDied","Data":"ec967ca957f5c9b0f1e5b013a78fd3f921d9617a5de282e8056425bc73a16453"}
Oct 30 22:05:09.218500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:09.218417    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5619f6cb-592c-47c1-9bfa-d9f10ff261a0" path="/var/lib/kubelet/pods/5619f6cb-592c-47c1-9bfa-d9f10ff261a0/volumes"
Oct 30 22:05:10.312569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.312527    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:10.440453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.440341    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca\") pod \"005a4540-df13-41cb-8371-3820f879b8bf\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") "
Oct 30 22:05:10.440706 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.440516    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt\" (UniqueName: \"kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt\") pod \"005a4540-df13-41cb-8371-3820f879b8bf\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") "
Oct 30 22:05:10.440706 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.440587    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca\") pod \"005a4540-df13-41cb-8371-3820f879b8bf\" (UID: \"005a4540-df13-41cb-8371-3820f879b8bf\") "
Oct 30 22:05:10.440868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.440813    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca" pod "005a4540-df13-41cb-8371-3820f879b8bf" (UID: "005a4540-df13-41cb-8371-3820f879b8bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:10.441281 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.441252    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:10.443046 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.443007    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca" pod "005a4540-df13-41cb-8371-3820f879b8bf" (UID: "005a4540-df13-41cb-8371-3820f879b8bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:10.443046 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.443012    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt" pod "005a4540-df13-41cb-8371-3820f879b8bf" (UID: "005a4540-df13-41cb-8371-3820f879b8bf"). InnerVolumeSpecName "kube-api-access-rdpmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:10.543191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.543142    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpmt\" (UniqueName: \"kubernetes.io/projected/005a4540-df13-41cb-8371-3820f879b8bf-kube-api-access-rdpmt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:10.543191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.543183    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005a4540-df13-41cb-8371-3820f879b8bf-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:10.807485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.807437    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"]
Oct 30 22:05:10.842632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:10.842570    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"]
Oct 30 22:05:11.125717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:11.125628    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"
Oct 30 22:05:11.125717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:11.125635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7" event={"ID":"005a4540-df13-41cb-8371-3820f879b8bf","Type":"ContainerDied","Data":"c2b29cb984aa845bbcc0e229b3800c9d41a4d6eb43508627584104f351d1cf61"}
Oct 30 22:05:11.125717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:11.125693    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2b29cb984aa845bbcc0e229b3800c9d41a4d6eb43508627584104f351d1cf61"
Oct 30 22:05:15.808711 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:15.808660    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr" podUID="da1396da-8dc6-4dd9-b63e-325dd0e1bf32" containerName="payload-processing" containerID="cri-o://4ae576d1ea323bf0c1a9d3077e906cb5658d62ebc54dde8de52d6abf274e74c6" gracePeriod=25
Oct 30 22:05:15.843459 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:15.843407    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerName="payload-pre-processing" containerID="cri-o://c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d" gracePeriod=25
Oct 30 22:05:16.397456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.395672    6695 generic.go:356] "Generic (PLEG): container finished" podID="da1396da-8dc6-4dd9-b63e-325dd0e1bf32" containerID="4ae576d1ea323bf0c1a9d3077e906cb5658d62ebc54dde8de52d6abf274e74c6" exitCode=0
Oct 30 22:05:16.397456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.395874    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr" event={"ID":"da1396da-8dc6-4dd9-b63e-325dd0e1bf32","Type":"ContainerDied","Data":"4ae576d1ea323bf0c1a9d3077e906cb5658d62ebc54dde8de52d6abf274e74c6"}
Oct 30 22:05:16.521543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.521505    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 22:05:16.601179 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.601135    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987\" (UniqueName: \"kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987\") pod \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") "
Oct 30 22:05:16.601480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.601462    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume\") pod \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\" (UID: \"da1396da-8dc6-4dd9-b63e-325dd0e1bf32\") "
Oct 30 22:05:16.601991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.601947    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume" pod "da1396da-8dc6-4dd9-b63e-325dd0e1bf32" (UID: "da1396da-8dc6-4dd9-b63e-325dd0e1bf32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:16.602254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.602234    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:16.603707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.603673    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987" pod "da1396da-8dc6-4dd9-b63e-325dd0e1bf32" (UID: "da1396da-8dc6-4dd9-b63e-325dd0e1bf32"). InnerVolumeSpecName "kube-api-access-ml987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:16.704470 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:16.704373    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml987\" (UniqueName: \"kubernetes.io/projected/da1396da-8dc6-4dd9-b63e-325dd0e1bf32-kube-api-access-ml987\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:17.407130 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:17.407096    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"
Oct 30 22:05:17.407700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:17.407129    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr" event={"ID":"da1396da-8dc6-4dd9-b63e-325dd0e1bf32","Type":"ContainerDied","Data":"02165e63d691ca77e97d362ada8c0b170595ec6ac033cc047a03843e4023b4a4"}
Oct 30 22:05:17.407700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:17.407203    6695 scope.go:122] "RemoveContainer" containerID="4ae576d1ea323bf0c1a9d3077e906cb5658d62ebc54dde8de52d6abf274e74c6"
Oct 30 22:05:17.464829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:17.464789    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"]
Oct 30 22:05:17.476408 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:17.476363    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-db1288-5dd6b4bf99-qdmhr"]
Oct 30 22:05:19.247517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.247480    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1396da-8dc6-4dd9-b63e-325dd0e1bf32" path="/var/lib/kubelet/pods/da1396da-8dc6-4dd9-b63e-325dd0e1bf32/volumes"
Oct 30 22:05:19.668951 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.668912    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"]
Oct 30 22:05:19.670663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.670002    6695 cpu_manager.go:414] "containerMap: removing container" podUID="005a4540-df13-41cb-8371-3820f879b8bf" containerName="revoke-keys"
Oct 30 22:05:19.670663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.670023    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005a4540-df13-41cb-8371-3820f879b8bf" containerName="revoke-keys"
Oct 30 22:05:19.670663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.670047    6695 cpu_manager.go:414] "containerMap: removing container" podUID="da1396da-8dc6-4dd9-b63e-325dd0e1bf32" containerName="payload-processing"
Oct 30 22:05:19.670663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.670053    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1396da-8dc6-4dd9-b63e-325dd0e1bf32" containerName="payload-processing"
Oct 30 22:05:19.670663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.670171    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="005a4540-df13-41cb-8371-3820f879b8bf" containerName="revoke-keys"
Oct 30 22:05:19.670663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.670182    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1396da-8dc6-4dd9-b63e-325dd0e1bf32" containerName="payload-processing"
Oct 30 22:05:19.675521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.675497    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.693207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.693179    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"]
Oct 30 22:05:19.712593 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.712535    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.713207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.713160    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.713483 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.713461    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfj5\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.713789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.713768    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.828796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.828741    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.832311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.832271    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.832756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.832617    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfj5\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.833066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.833044    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.834481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.834454    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.835043 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.830212    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.841886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.841845    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:19.855201 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:19.855165    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfj5\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:20.015012 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.014902    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:20.020419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.020385    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"]
Oct 30 22:05:20.024978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.024939    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh" podUID="cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" containerName="sim" containerID="cri-o://d90ce44e0f712ba9120313162685846cc04c6850fed55e272a4322ea3251dc54" gracePeriod=30
Oct 30 22:05:20.551629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.550941    6695 generic.go:356] "Generic (PLEG): container finished" podID="cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" containerID="d90ce44e0f712ba9120313162685846cc04c6850fed55e272a4322ea3251dc54" exitCode=0
Oct 30 22:05:20.551629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.551038    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh" event={"ID":"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6","Type":"ContainerDied","Data":"d90ce44e0f712ba9120313162685846cc04c6850fed55e272a4322ea3251dc54"}
Oct 30 22:05:20.668811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.668420    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"]
Oct 30 22:05:20.692099 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.692013    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 22:05:20.696996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.696957    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.700537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.700501    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"]
Oct 30 22:05:20.705686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.705659    6695 kubelet.go:1689] "Image garbage collection succeeded"
Oct 30 22:05:20.709491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.709456    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 22:05:20.753214 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.753178    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-e2e--68c3d057-dockercfg-2s4kt\""
Oct 30 22:05:20.753423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.753358    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"kube-root-ca.crt\""
Oct 30 22:05:20.753644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.753587    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ed0866\""
Oct 30 22:05:20.753862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.753844    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"openshift-service-ca.crt\""
Oct 30 22:05:20.754286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.754070    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"routing-overlay\""
Oct 30 22:05:20.754391 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.754140    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-ed981606a7c4\""
Oct 30 22:05:20.881061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.880658    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.881281 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.881199    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.881341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.881279    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.894800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.881449    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.907351 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:20.907311    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5645c677_ea16_4ddc_ab56_414809833f94.slice/crio-73347bfd4ee64cee16e62bad434a43803238da1ea29ae8114e5ab9409041cbf9 WatchSource:0}: Error finding container 73347bfd4ee64cee16e62bad434a43803238da1ea29ae8114e5ab9409041cbf9: Status 404 returned error can't find the container with id 73347bfd4ee64cee16e62bad434a43803238da1ea29ae8114e5ab9409041cbf9
Oct 30 22:05:20.943707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.940862    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"]
Oct 30 22:05:20.957336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.957302    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:05:20.989301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.989262    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm\" (UniqueName: \"kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm\") pod \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") "
Oct 30 22:05:20.989588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.989563    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls\" (UniqueName: \"kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls\") pod \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\" (UID: \"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6\") "
Oct 30 22:05:20.990996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.990092    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.990996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.990157    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.990996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.990304    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.990996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.990407    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.991360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.991227    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.991360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.991272    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:20.992082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.992049    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm" pod "cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" (UID: "cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6"). InnerVolumeSpecName "kube-api-access-8d8bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:20.992233 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.992175    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls" pod "cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" (UID: "cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:20.993340 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:20.993317    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:21.034945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.010976    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:21.049345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.049313    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:21.094300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.092736    6695 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-tls\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:21.094300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.092777    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8bm\" (UniqueName: \"kubernetes.io/projected/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6-kube-api-access-8d8bm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:21.158762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.158724    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:05:21.394842 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.394810    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"]
Oct 30 22:05:21.397476 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:21.397438    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0fb35f_a0ef_4bfe_a5de_3257a0668e70.slice/crio-9905f284f8f5a1e214c710f6f56afa30cae483db2fb4ff37ead5055d92f4da91 WatchSource:0}: Error finding container 9905f284f8f5a1e214c710f6f56afa30cae483db2fb4ff37ead5055d92f4da91: Status 404 returned error can't find the container with id 9905f284f8f5a1e214c710f6f56afa30cae483db2fb4ff37ead5055d92f4da91
Oct 30 22:05:21.564064 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.563958    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" event={"ID":"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70","Type":"ContainerStarted","Data":"c85fdd191b1948528440692c03f5ea2dab75f964ccc9d297667d6a6328e42acb"}
Oct 30 22:05:21.564064 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.564025    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" event={"ID":"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70","Type":"ContainerStarted","Data":"9905f284f8f5a1e214c710f6f56afa30cae483db2fb4ff37ead5055d92f4da91"}
Oct 30 22:05:21.564627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.564094    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:21.566101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.566066    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" event={"ID":"5645c677-ea16-4ddc-ab56-414809833f94","Type":"ContainerStarted","Data":"092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99"}
Oct 30 22:05:21.566266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.566130    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" event={"ID":"5645c677-ea16-4ddc-ab56-414809833f94","Type":"ContainerStarted","Data":"73347bfd4ee64cee16e62bad434a43803238da1ea29ae8114e5ab9409041cbf9"}
Oct 30 22:05:21.566266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.566158    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:21.568001 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.567967    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh" event={"ID":"cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6","Type":"ContainerDied","Data":"8263c4bc350868ccea3dab4f7e3acee64a561376140a76673fbeeff5739fe129"}
Oct 30 22:05:21.568001 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.567987    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"
Oct 30 22:05:21.568363 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.568026    6695 scope.go:122] "RemoveContainer" containerID="d90ce44e0f712ba9120313162685846cc04c6850fed55e272a4322ea3251dc54"
Oct 30 22:05:21.601080 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:21.600973    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5d3601/agc-cred-3018d9e8cbed: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:21.601080 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:21.601012    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:21.601369 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:21.601088    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials podName:5645c677-ea16-4ddc-ab56-414809833f94 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:22.101060796 +0000 UTC m=+3305.023519087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" (UID: "5645c677-ea16-4ddc-ab56-414809833f94") : secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:21.608734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.608662    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" podStartSLOduration=1.608640819 podStartE2EDuration="1.608640819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:21.604591296 +0000 UTC m=+3304.527049594" watchObservedRunningTime="2026-10-30 22:05:21.608640819 +0000 UTC m=+3304.531099120"
Oct 30 22:05:21.659553 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.659486    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" podStartSLOduration=2.659471043 podStartE2EDuration="2.659471043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:21.658893441 +0000 UTC m=+3304.581351740" watchObservedRunningTime="2026-10-30 22:05:21.659471043 +0000 UTC m=+3304.581929340"
Oct 30 22:05:21.697236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.697191    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"]
Oct 30 22:05:21.704932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:21.704889    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-4b2713/e2e-usage-sim-cea690-594dc6847c-ptbxh"]
Oct 30 22:05:22.118630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:22.115533    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"]
Oct 30 22:05:22.141630 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.127298    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5d3601/agc-cred-3018d9e8cbed: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:22.141630 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.127336    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:22.141630 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.127402    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials podName:5645c677-ea16-4ddc-ab56-414809833f94 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:23.127380383 +0000 UTC m=+3306.049838676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" (UID: "5645c677-ea16-4ddc-ab56-414809833f94") : secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:22.672685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:22.665383    6695 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" secret="" err="secret \"payload-processing-external-model-e2e--68c3d057-dockercfg-2s4kt\" not found"
Oct 30 22:05:22.746162 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.746113    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-plugins-e2e-extmod-b-ed0866: configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:22.746412 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.746228    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:23.246200956 +0000 UTC m=+3306.168659291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:22.746412 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.746296    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ed0866/agc-cred-ed981606a7c4: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:22.746412 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.746322    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:22.746412 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:22.746388    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:23.246370502 +0000 UTC m=+3306.168828785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:23.164625 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.164172    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5d3601/agc-cred-3018d9e8cbed: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:23.164625 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.164208    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:23.164625 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.164271    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials podName:5645c677-ea16-4ddc-ab56-414809833f94 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:25.164250521 +0000 UTC m=+3308.086708800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" (UID: "5645c677-ea16-4ddc-ab56-414809833f94") : secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:23.232451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:23.232406    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" path="/var/lib/kubelet/pods/cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6/volumes"
Oct 30 22:05:23.266281 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.266237    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-plugins-e2e-extmod-b-ed0866: configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:23.266501 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.266308    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ed0866/agc-cred-ed981606a7c4: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:23.266501 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.266331    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:23.266501 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.266343    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:24.266316568 +0000 UTC m=+3307.188774858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:23.266501 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:23.266369    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:24.266356475 +0000 UTC m=+3307.188814752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:24.013394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:24.012925    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"]
Oct 30 22:05:24.018271 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:24.018230    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"]
Oct 30 22:05:24.285064 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:24.285021    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-plugins-e2e-extmod-b-ed0866: configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:24.285287 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:24.285115    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:26.285093091 +0000 UTC m=+3309.207551368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:24.285287 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:24.285175    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ed0866/agc-cred-ed981606a7c4: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:24.285287 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:24.285198    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:24.285287 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:24.285252    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:26.285237669 +0000 UTC m=+3309.207695967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:25.203176 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:25.203128    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5d3601/agc-cred-3018d9e8cbed: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:25.203176 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:25.203171    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:25.203772 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:25.203226    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials podName:5645c677-ea16-4ddc-ab56-414809833f94 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:29.203212408 +0000 UTC m=+3312.125670684 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" (UID: "5645c677-ea16-4ddc-ab56-414809833f94") : secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:26.323408 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:26.323347    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-ed0866/agc-cred-ed981606a7c4: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:26.323408 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:26.323385    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n: secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:26.324179 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:26.323447    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:30.323429094 +0000 UTC m=+3313.245887390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : secret "agc-cred-ed981606a7c4" not found
Oct 30 22:05:26.324179 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:26.323468    6695 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-plugins-e2e-extmod-b-ed0866: configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:26.324179 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:26.323508    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume podName:4d0fb35f-a0ef-4bfe-a5de-3257a0668e70 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:30.323498951 +0000 UTC m=+3313.245957242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-ed0866" not found
Oct 30 22:05:27.294956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.294898    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"]
Oct 30 22:05:27.296402 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.296369    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" containerName="sim"
Oct 30 22:05:27.297178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.297152    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" containerName="sim"
Oct 30 22:05:27.297688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.297662    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5ea70b-623d-4eda-b7cc-188cf3fe1fd6" containerName="sim"
Oct 30 22:05:27.335037 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.334989    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"]
Oct 30 22:05:27.335617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.335138    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.342236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.342204    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.342438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.342386    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.342739 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.342704    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.343306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.342937    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.343306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.342981    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.343306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.343083    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.343306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.343157    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.343306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.343224    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hz8\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.344025 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.343881    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.364965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.364926    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6537e6-openshift-default-dockercfg-2bx9p\""
Oct 30 22:05:27.445003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.444962    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445074    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445098    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445154    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445204    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445247    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hz8\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445345    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445384    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445430    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445835    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.445961 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.445873    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.446156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.446135    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.446244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.446219    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.446286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.446269    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.447754 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.447730    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.447899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.447884    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.469089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.469055    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.469269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.469095    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hz8\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8\") pod \"e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") " pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:27.767679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:27.764053    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:28.166109 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.166079    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"]
Oct 30 22:05:28.169664 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:28.169633    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a33640_8558_4199_b50c_2c64140f338a.slice/crio-48a6b4a8da0f72f258a99f1fb74f73c22ec55c5ab74eefb392d82b1ab9c1295d WatchSource:0}: Error finding container 48a6b4a8da0f72f258a99f1fb74f73c22ec55c5ab74eefb392d82b1ab9c1295d: Status 404 returned error can't find the container with id 48a6b4a8da0f72f258a99f1fb74f73c22ec55c5ab74eefb392d82b1ab9c1295d
Oct 30 22:05:28.173470 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.173433    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:28.173573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.173509    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:28.173573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.173543    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:28.675712 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.675668    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" podUID="4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" containerName="payload-processing" containerID="cri-o://c85fdd191b1948528440692c03f5ea2dab75f964ccc9d297667d6a6328e42acb" gracePeriod=25
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.801730    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.801787    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.825584    6695 generic.go:356] "Generic (PLEG): container finished" podID="4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" containerID="c85fdd191b1948528440692c03f5ea2dab75f964ccc9d297667d6a6328e42acb" exitCode=0
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.825754    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" event={"ID":"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70","Type":"ContainerDied","Data":"c85fdd191b1948528440692c03f5ea2dab75f964ccc9d297667d6a6328e42acb"}
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.827547    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" event={"ID":"79a33640-8558-4199-b50c-2c64140f338a","Type":"ContainerStarted","Data":"7896cbd2b09f2af560736ccc12171e0ed7c021e02a97c512993ff0badf5d87dc"}
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.827643    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" event={"ID":"79a33640-8558-4199-b50c-2c64140f338a","Type":"ContainerStarted","Data":"48a6b4a8da0f72f258a99f1fb74f73c22ec55c5ab74eefb392d82b1ab9c1295d"}
Oct 30 22:05:29.017510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:28.878652    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podStartSLOduration=1.878631034 podStartE2EDuration="1.878631034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:28.873779869 +0000 UTC m=+3311.796238168" watchObservedRunningTime="2026-10-30 22:05:28.878631034 +0000 UTC m=+3311.801089333"
Oct 30 22:05:29.052415 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.020682    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx" podUID="9c336a4c-d2a7-459f-b1e9-a2f79109c186" containerName="payload-processing" containerID="cri-o://6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34" gracePeriod=25
Oct 30 22:05:29.052415 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.021030    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" podUID="5645c677-ea16-4ddc-ab56-414809833f94" containerName="payload-processing" containerID="cri-o://092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99" gracePeriod=25
Oct 30 22:05:29.138556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.132503    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.299045    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume\") pod \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") "
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.299118    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs\") pod \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") "
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.299431    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials\") pod \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") "
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.299649    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume\") pod \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\" (UID: \"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70\") "
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:29.300773    6695 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-5d3601/agc-cred-3018d9e8cbed: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:29.300797    6695 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g: secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:29.301624 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:29.300905    6695 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials podName:5645c677-ea16-4ddc-ab56-414809833f94 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:37.300884282 +0000 UTC m=+3320.223342560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" (UID: "5645c677-ea16-4ddc-ab56-414809833f94") : secret "agc-cred-3018d9e8cbed" not found
Oct 30 22:05:29.307167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.305013    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume" pod "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:29.307725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.307643    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume" pod "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:29.326230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.326184    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials" pod "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.340085 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.339792    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs" pod "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" (UID: "4d0fb35f-a0ef-4bfe-a5de-3257a0668e70"). InnerVolumeSpecName "kube-api-access-nq7gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.369940 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:29.369899    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c336a4c_d2a7_459f_b1e9_a2f79109c186.slice/crio-6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:05:29.402519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.402480    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.402519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.402514    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.402519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.402523    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.402519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.402532    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq7gs\" (UniqueName: \"kubernetes.io/projected/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70-kube-api-access-nq7gs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.600766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.600726    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:29.605206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.605175    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:05:29.605701 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.605676    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume\") pod \"5645c677-ea16-4ddc-ab56-414809833f94\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") "
Oct 30 22:05:29.605776 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.605729    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume\") pod \"5645c677-ea16-4ddc-ab56-414809833f94\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") "
Oct 30 22:05:29.605920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.605903    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5\") pod \"5645c677-ea16-4ddc-ab56-414809833f94\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") "
Oct 30 22:05:29.606153 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.606131    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume" pod "5645c677-ea16-4ddc-ab56-414809833f94" (UID: "5645c677-ea16-4ddc-ab56-414809833f94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:29.606245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.606128    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume" pod "5645c677-ea16-4ddc-ab56-414809833f94" (UID: "5645c677-ea16-4ddc-ab56-414809833f94"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:29.606245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.606222    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials\") pod \"5645c677-ea16-4ddc-ab56-414809833f94\" (UID: \"5645c677-ea16-4ddc-ab56-414809833f94\") "
Oct 30 22:05:29.606836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.606813    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.606913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.606838    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5645c677-ea16-4ddc-ab56-414809833f94-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.608290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.608260    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5" pod "5645c677-ea16-4ddc-ab56-414809833f94" (UID: "5645c677-ea16-4ddc-ab56-414809833f94"). InnerVolumeSpecName "kube-api-access-hzfj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.608420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.608370    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials" pod "5645c677-ea16-4ddc-ab56-414809833f94" (UID: "5645c677-ea16-4ddc-ab56-414809833f94"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.707895 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.707857    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume\") pod \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") "
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.708005    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd\") pod \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") "
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.708083    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume\") pod \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") "
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.708186    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials\") pod \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\" (UID: \"9c336a4c-d2a7-459f-b1e9-a2f79109c186\") "
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.708314    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume" pod "9c336a4c-d2a7-459f-b1e9-a2f79109c186" (UID: "9c336a4c-d2a7-459f-b1e9-a2f79109c186"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.708752    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume" pod "9c336a4c-d2a7-459f-b1e9-a2f79109c186" (UID: "9c336a4c-d2a7-459f-b1e9-a2f79109c186"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.709103    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.709127    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfj5\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-kube-api-access-hzfj5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.709142    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c336a4c-d2a7-459f-b1e9-a2f79109c186-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.709869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.709155    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5645c677-ea16-4ddc-ab56-414809833f94-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.711407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.711377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials" pod "9c336a4c-d2a7-459f-b1e9-a2f79109c186" (UID: "9c336a4c-d2a7-459f-b1e9-a2f79109c186"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.713091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.713058    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd" pod "9c336a4c-d2a7-459f-b1e9-a2f79109c186" (UID: "9c336a4c-d2a7-459f-b1e9-a2f79109c186"). InnerVolumeSpecName "kube-api-access-nnshd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:29.765543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.765505    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:29.772533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.772499    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:29.811087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.810987    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnshd\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-kube-api-access-nnshd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.811087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.811026    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c336a4c-d2a7-459f-b1e9-a2f79109c186-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:29.838432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.838395    6695 generic.go:356] "Generic (PLEG): container finished" podID="5645c677-ea16-4ddc-ab56-414809833f94" containerID="092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99" exitCode=0
Oct 30 22:05:29.838660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.838445    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"
Oct 30 22:05:29.838660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.838484    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" event={"ID":"5645c677-ea16-4ddc-ab56-414809833f94","Type":"ContainerDied","Data":"092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99"}
Oct 30 22:05:29.838660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.838543    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g" event={"ID":"5645c677-ea16-4ddc-ab56-414809833f94","Type":"ContainerDied","Data":"73347bfd4ee64cee16e62bad434a43803238da1ea29ae8114e5ab9409041cbf9"}
Oct 30 22:05:29.838660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.838574    6695 scope.go:122] "RemoveContainer" containerID="092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99"
Oct 30 22:05:29.840152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.840128    6695 generic.go:356] "Generic (PLEG): container finished" podID="9c336a4c-d2a7-459f-b1e9-a2f79109c186" containerID="6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34" exitCode=0
Oct 30 22:05:29.840263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.840192    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"
Oct 30 22:05:29.840263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.840214    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx" event={"ID":"9c336a4c-d2a7-459f-b1e9-a2f79109c186","Type":"ContainerDied","Data":"6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34"}
Oct 30 22:05:29.840376 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.840269    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx" event={"ID":"9c336a4c-d2a7-459f-b1e9-a2f79109c186","Type":"ContainerDied","Data":"f9be71f02b16e987b9562c40952335efc607c61ef158e9370eb12bcd5bdb9390"}
Oct 30 22:05:29.842677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.842651    6695 generic.go:356] "Generic (PLEG): container finished" podID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerID="1be1b9ec6a125ff27eabbc28741dfe24fa85b33b13a8de6a4f7a02e5c69c7c6a" exitCode=0
Oct 30 22:05:29.842835 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.842725    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" event={"ID":"49803228-087d-4b4c-8d79-fcc236abe8f8","Type":"ContainerDied","Data":"1be1b9ec6a125ff27eabbc28741dfe24fa85b33b13a8de6a4f7a02e5c69c7c6a"}
Oct 30 22:05:29.843236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.843217    6695 scope.go:122] "RemoveContainer" containerID="1be1b9ec6a125ff27eabbc28741dfe24fa85b33b13a8de6a4f7a02e5c69c7c6a"
Oct 30 22:05:29.845339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.845309    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:29.845446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.845398    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:29.845525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.845452    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:29.845525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.845490    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"
Oct 30 22:05:29.846743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.846549    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n" event={"ID":"4d0fb35f-a0ef-4bfe-a5de-3257a0668e70","Type":"ContainerDied","Data":"9905f284f8f5a1e214c710f6f56afa30cae483db2fb4ff37ead5055d92f4da91"}
Oct 30 22:05:29.846743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.846621    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:29.847052 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.847032    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:05:29.852152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.852112    6695 scope.go:122] "RemoveContainer" containerID="092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99"
Oct 30 22:05:29.852528 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:29.852497    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99\": container with ID starting with 092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99 not found: ID does not exist" containerID="092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99"
Oct 30 22:05:29.852672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.852539    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99"} err="failed to get container status \"092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99\": rpc error: code = NotFound desc = could not find container \"092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99\": container with ID starting with 092a8c2dd5702de2c9123f3f3410a90a52a80da52ea522158ca65cc58abbfe99 not found: ID does not exist"
Oct 30 22:05:29.852672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.852563    6695 scope.go:122] "RemoveContainer" containerID="6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34"
Oct 30 22:05:29.864661 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.864586    6695 scope.go:122] "RemoveContainer" containerID="6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34"
Oct 30 22:05:29.865129 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:29.865098    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34\": container with ID starting with 6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34 not found: ID does not exist" containerID="6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34"
Oct 30 22:05:29.865238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.865138    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34"} err="failed to get container status \"6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34\": rpc error: code = NotFound desc = could not find container \"6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34\": container with ID starting with 6580451b9e3416dcc858cc00de6c1895ac6d35686ce0e6c6c80d6f2fb0999f34 not found: ID does not exist"
Oct 30 22:05:29.865238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:29.865162    6695 scope.go:122] "RemoveContainer" containerID="c85fdd191b1948528440692c03f5ea2dab75f964ccc9d297667d6a6328e42acb"
Oct 30 22:05:30.021990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.021942    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"]
Oct 30 22:05:30.026847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.026797    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-6c5fllghx"]
Oct 30 22:05:30.063983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.063841    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"]
Oct 30 22:05:30.069072 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.069039    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-64b5dvl9n"]
Oct 30 22:05:30.102520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.102483    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"]
Oct 30 22:05:30.109186 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.109150    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-76f5ksh2g"]
Oct 30 22:05:30.370230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.370137    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"]
Oct 30 22:05:30.371900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.371865    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" containerName="payload-processing"
Oct 30 22:05:30.372156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.372138    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" containerName="payload-processing"
Oct 30 22:05:30.372274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.372263    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9c336a4c-d2a7-459f-b1e9-a2f79109c186" containerName="payload-processing"
Oct 30 22:05:30.372354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.372342    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c336a4c-d2a7-459f-b1e9-a2f79109c186" containerName="payload-processing"
Oct 30 22:05:30.372448 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.372439    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5645c677-ea16-4ddc-ab56-414809833f94" containerName="payload-processing"
Oct 30 22:05:30.372533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.372523    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5645c677-ea16-4ddc-ab56-414809833f94" containerName="payload-processing"
Oct 30 22:05:30.372925 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.372908    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" containerName="payload-processing"
Oct 30 22:05:30.373033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.373023    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c336a4c-d2a7-459f-b1e9-a2f79109c186" containerName="payload-processing"
Oct 30 22:05:30.373131 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.373121    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5645c677-ea16-4ddc-ab56-414809833f94" containerName="payload-processing"
Oct 30 22:05:30.403671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.399054    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.405992 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.405855    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"]
Oct 30 22:05:30.423957 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.423926    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5454af-openshift-default-dockercfg-gdnlh\""
Oct 30 22:05:30.436507 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.436478    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.436879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.436845    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.436951 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.436930    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.437523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.437003    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.437523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.437099    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.437523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.437168    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.437523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.437199    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfpl\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.437523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.437245    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.437523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.437299    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571254    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571344    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571422    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571485    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571773 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571558    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571773 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571623    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfpl\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571773 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571686    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.571773 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571748    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.572022 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.571900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.575284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.575253    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.577668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.576349    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.577668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.576675    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.577876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.577845    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.579036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.579014    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.580046 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.579968    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.582958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.582934    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.597684 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.597110    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.603291 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.603260    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfpl\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl\") pod \"e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") " pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.739691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.739215    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:30.919459 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:30.919419    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" event={"ID":"49803228-087d-4b4c-8d79-fcc236abe8f8","Type":"ContainerStarted","Data":"03da5571efda7cfa88311b6d442a2ad18427e8a48775230dc5652c545b9fb0d6"}
Oct 30 22:05:31.091108 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:31.091065    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:05:31.156963 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.156923    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:05:31.339441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.339072    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:31.339441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.339246    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:31.339441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.339338    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:31.359781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.359551    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0fb35f-a0ef-4bfe-a5de-3257a0668e70" path="/var/lib/kubelet/pods/4d0fb35f-a0ef-4bfe-a5de-3257a0668e70/volumes"
Oct 30 22:05:31.360342 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.360311    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5645c677-ea16-4ddc-ab56-414809833f94" path="/var/lib/kubelet/pods/5645c677-ea16-4ddc-ab56-414809833f94/volumes"
Oct 30 22:05:31.360902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.360876    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c336a4c-d2a7-459f-b1e9-a2f79109c186" path="/var/lib/kubelet/pods/9c336a4c-d2a7-459f-b1e9-a2f79109c186/volumes"
Oct 30 22:05:31.361325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.361306    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"]
Oct 30 22:05:31.629687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.616016    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:05:31.629687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.616144    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:31.629687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.616187    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused"
Oct 30 22:05:31.948983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.948893    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" event={"ID":"3c2f7991-55c5-4109-b142-d8371f73f09c","Type":"ContainerStarted","Data":"2e32d1804e8470755d00c3996eb58f19cbe8e8e57a4de5808503c5974fb8e3f5"}
Oct 30 22:05:31.949498 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:31.949473    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" event={"ID":"3c2f7991-55c5-4109-b142-d8371f73f09c","Type":"ContainerStarted","Data":"72f0ccaaa0e2724f190f25c4b7884a6a99155ca948e03035eb5a818b5f34a05c"}
Oct 30 22:05:32.001231 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.001173    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podStartSLOduration=2.001156818 podStartE2EDuration="2.001156818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:31.997076278 +0000 UTC m=+3314.919534577" watchObservedRunningTime="2026-10-30 22:05:32.001156818 +0000 UTC m=+3314.923615128"
Oct 30 22:05:32.623629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.614978    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:32.623629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.615043    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused"
Oct 30 22:05:32.740007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.739965    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:32.747310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.747277    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:32.962012 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.961925    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:32.963087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:32.963060    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:05:33.360094 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.360050    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"]
Oct 30 22:05:33.365177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.365153    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.388726 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.388691    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6537e6-dockercfg-zljnp\""
Oct 30 22:05:33.393017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.392983    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6537e6\""
Oct 30 22:05:33.395274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.395241    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"]
Oct 30 22:05:33.521639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.519076    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277m9\" (UniqueName: \"kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9\") pod \"payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.521639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.519127    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.620762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.617397    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:33.620762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.617459    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused"
Oct 30 22:05:33.733045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.724431    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277m9\" (UniqueName: \"kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9\") pod \"payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.733045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.724482    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.736456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.734349    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.778967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.778916    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"]
Oct 30 22:05:33.783491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.783446    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-277m9\" (UniqueName: \"kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9\") pod \"payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:33.785159 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.785135    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.785341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.785316    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"]
Oct 30 22:05:33.839521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.839469    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktb9\" (UniqueName: \"kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9\") pod \"payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.839780 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.839762    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.940977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.940876    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.941172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.941037    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktb9\" (UniqueName: \"kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9\") pod \"payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.941643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.941586    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.963164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.963119    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktb9\" (UniqueName: \"kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9\") pod \"payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:33.983829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:33.983785    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:34.134635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:34.120250    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:34.348051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:34.347826    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"]
Oct 30 22:05:34.349533 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:34.349505    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197c2517_8f1a_47b6_ac8a_d42ae83e4b56.slice/crio-49c268633a668f3f8006b10110abed423799db2c09f801a36c17393a0123fe41 WatchSource:0}: Error finding container 49c268633a668f3f8006b10110abed423799db2c09f801a36c17393a0123fe41: Status 404 returned error can't find the container with id 49c268633a668f3f8006b10110abed423799db2c09f801a36c17393a0123fe41
Oct 30 22:05:34.505490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:34.505461    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"]
Oct 30 22:05:34.508423 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:34.508395    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187b23fa_08f8_4d55_8fe6_4ec57bc52824.slice/crio-460afa73beea1fb2a4c8a3267f857bc54838949a3801b8a4f9104f7526f2dd1c WatchSource:0}: Error finding container 460afa73beea1fb2a4c8a3267f857bc54838949a3801b8a4f9104f7526f2dd1c: Status 404 returned error can't find the container with id 460afa73beea1fb2a4c8a3267f857bc54838949a3801b8a4f9104f7526f2dd1c
Oct 30 22:05:34.638633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:34.637738    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:34.638633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:34.637800    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused"
Oct 30 22:05:35.006625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.005163    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s" event={"ID":"187b23fa-08f8-4d55-8fe6-4ec57bc52824","Type":"ContainerStarted","Data":"48dbaf4fe7f2f9e294a7304c2443d225ec2cfca93c544b89f57eb3cb2eec4176"}
Oct 30 22:05:35.006625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.005244    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s" event={"ID":"187b23fa-08f8-4d55-8fe6-4ec57bc52824","Type":"ContainerStarted","Data":"460afa73beea1fb2a4c8a3267f857bc54838949a3801b8a4f9104f7526f2dd1c"}
Oct 30 22:05:35.006625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.006545    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:35.015629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.013696    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" event={"ID":"197c2517-8f1a-47b6-ac8a-d42ae83e4b56","Type":"ContainerStarted","Data":"645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25"}
Oct 30 22:05:35.015629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.013769    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" event={"ID":"197c2517-8f1a-47b6-ac8a-d42ae83e4b56","Type":"ContainerStarted","Data":"49c268633a668f3f8006b10110abed423799db2c09f801a36c17393a0123fe41"}
Oct 30 22:05:35.022630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.017757    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:35.055631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.052354    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s" podStartSLOduration=2.05233326 podStartE2EDuration="2.05233326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:35.051029735 +0000 UTC m=+3317.973488035" watchObservedRunningTime="2026-10-30 22:05:35.05233326 +0000 UTC m=+3317.974791555"
Oct 30 22:05:35.099672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.099573    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" podStartSLOduration=2.09955817 podStartE2EDuration="2.09955817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:35.097924404 +0000 UTC m=+3318.020382701" watchObservedRunningTime="2026-10-30 22:05:35.09955817 +0000 UTC m=+3318.022016467"
Oct 30 22:05:35.641103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.641052    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:35.641338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:35.641116    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused"
Oct 30 22:05:36.243726 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:36.241068    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72107bfe_f7a3_4d3c_a808_7ebd99e1d809.slice/crio-conmon-c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:05:36.393725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.393694    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 22:05:36.481909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.481864    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume\") pod \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") "
Oct 30 22:05:36.482094 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.482049    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8\" (UniqueName: \"kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8\") pod \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\" (UID: \"72107bfe-f7a3-4d3c-a808-7ebd99e1d809\") "
Oct 30 22:05:36.483575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.483544    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume" pod "72107bfe-f7a3-4d3c-a808-7ebd99e1d809" (UID: "72107bfe-f7a3-4d3c-a808-7ebd99e1d809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:36.485017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.484991    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8" pod "72107bfe-f7a3-4d3c-a808-7ebd99e1d809" (UID: "72107bfe-f7a3-4d3c-a808-7ebd99e1d809"). InnerVolumeSpecName "kube-api-access-9kbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:36.584792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.584753    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:36.584977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.584793    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kbk8\" (UniqueName: \"kubernetes.io/projected/72107bfe-f7a3-4d3c-a808-7ebd99e1d809-kube-api-access-9kbk8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:36.641997 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.641948    6695 patch_prober.go:29] interesting pod/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:36.642208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.642003    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.169:15021/healthz/ready\": dial tcp 10.134.1.169:15021: connect: connection refused"
Oct 30 22:05:36.744260 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.744220    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"]
Oct 30 22:05:36.745219 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.745194    6695 cpu_manager.go:414] "containerMap: removing container" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerName="payload-pre-processing"
Oct 30 22:05:36.745219 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.745218    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerName="payload-pre-processing"
Oct 30 22:05:36.745455 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.745442    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerName="payload-pre-processing"
Oct 30 22:05:36.749731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.749709    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:36.766894 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.766858    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5454af\""
Oct 30 22:05:36.767099 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.767082    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5454af-dockercfg-sqdsb\""
Oct 30 22:05:36.770115 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.770062    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"]
Oct 30 22:05:36.956667 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:36.894562    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.011948 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.011844    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x676x\" (UniqueName: \"kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x\") pod \"payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.036377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.006766    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"]
Oct 30 22:05:37.113181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.113141    6695 generic.go:356] "Generic (PLEG): container finished" podID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" containerID="c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d" exitCode=0
Oct 30 22:05:37.113447 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.113420    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"
Oct 30 22:05:37.114869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.113279    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"]
Oct 30 22:05:37.115145 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.115116    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" event={"ID":"72107bfe-f7a3-4d3c-a808-7ebd99e1d809","Type":"ContainerDied","Data":"c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d"}
Oct 30 22:05:37.115299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.115281    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2" event={"ID":"72107bfe-f7a3-4d3c-a808-7ebd99e1d809","Type":"ContainerDied","Data":"a90e18ccd7e4f39d660e3d08db402db24c199a1770055ef48c20061d9abb58a8"}
Oct 30 22:05:37.115452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.115437    6695 scope.go:122] "RemoveContainer" containerID="c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d"
Oct 30 22:05:37.116069 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.113365    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.130071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.130031    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8c2m\" (UniqueName: \"kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m\") pod \"payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.130260 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.130238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.130381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.130365    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x676x\" (UniqueName: \"kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x\") pod \"payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.130452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.130428    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.132262 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.131811    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.145589 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.145558    6695 scope.go:122] "RemoveContainer" containerID="c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d"
Oct 30 22:05:37.146134 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:05:37.146091    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d\": container with ID starting with c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d not found: ID does not exist" containerID="c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d"
Oct 30 22:05:37.146232 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.146167    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d"} err="failed to get container status \"c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d\": rpc error: code = NotFound desc = could not find container \"c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d\": container with ID starting with c5b1a18a3ab94ccdfcbb6e3620c23a74f776074f3d08e65e3de3c9c310c0ee4d not found: ID does not exist"
Oct 30 22:05:37.150452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.150418    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x676x\" (UniqueName: \"kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x\") pod \"payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.219190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.219157    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"]
Oct 30 22:05:37.219190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.219200    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-db1288-6659f4fc5f-kmkc2"]
Oct 30 22:05:37.232431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.232389    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.233306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.232818    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8c2m\" (UniqueName: \"kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m\") pod \"payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.233306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.233296    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.252026 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.251986    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8c2m\" (UniqueName: \"kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m\") pod \"payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.366964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.366927    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:37.448308 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.448280    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:37.638671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.632997    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:05:37.638671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.634339    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:05:37.641967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.639046    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:05:37.752319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.751504    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"]
Oct 30 22:05:37.755484 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:37.755439    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e019eb_f65a_47af_afb9_5ec631bd285c.slice/crio-7b24564bb6e5171c1ce9151af859c34af86457b8d821dfd514cd33351b8aad8e WatchSource:0}: Error finding container 7b24564bb6e5171c1ce9151af859c34af86457b8d821dfd514cd33351b8aad8e: Status 404 returned error can't find the container with id 7b24564bb6e5171c1ce9151af859c34af86457b8d821dfd514cd33351b8aad8e
Oct 30 22:05:37.868883 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:37.868846    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"]
Oct 30 22:05:37.869748 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:37.869719    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda990936b_267c_4d6d_8a45_9728ca2d4e7d.slice/crio-beaeff6370d5fc77029c720090127cea032a0522d4971cb787453f10bb6bae99 WatchSource:0}: Error finding container beaeff6370d5fc77029c720090127cea032a0522d4971cb787453f10bb6bae99: Status 404 returned error can't find the container with id beaeff6370d5fc77029c720090127cea032a0522d4971cb787453f10bb6bae99
Oct 30 22:05:38.125249 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.125135    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl" event={"ID":"a990936b-267c-4d6d-8a45-9728ca2d4e7d","Type":"ContainerStarted","Data":"3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e"}
Oct 30 22:05:38.125249 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.125205    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl" event={"ID":"a990936b-267c-4d6d-8a45-9728ca2d4e7d","Type":"ContainerStarted","Data":"beaeff6370d5fc77029c720090127cea032a0522d4971cb787453f10bb6bae99"}
Oct 30 22:05:38.125249 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.125241    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:38.128088 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.128055    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" event={"ID":"a8e019eb-f65a-47af-afb9-5ec631bd285c","Type":"ContainerStarted","Data":"6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e"}
Oct 30 22:05:38.128235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.128113    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:38.128235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.128130    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" event={"ID":"a8e019eb-f65a-47af-afb9-5ec631bd285c","Type":"ContainerStarted","Data":"7b24564bb6e5171c1ce9151af859c34af86457b8d821dfd514cd33351b8aad8e"}
Oct 30 22:05:38.165786 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.165410    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl" podStartSLOduration=2.165390866 podStartE2EDuration="2.165390866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:38.162555639 +0000 UTC m=+3321.085013952" watchObservedRunningTime="2026-10-30 22:05:38.165390866 +0000 UTC m=+3321.087849164"
Oct 30 22:05:38.203405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:38.203350    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" podStartSLOduration=2.203336314 podStartE2EDuration="2.203336314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:38.198210367 +0000 UTC m=+3321.120668666" watchObservedRunningTime="2026-10-30 22:05:38.203336314 +0000 UTC m=+3321.125794638"
Oct 30 22:05:39.218666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:39.218626    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72107bfe-f7a3-4d3c-a808-7ebd99e1d809" path="/var/lib/kubelet/pods/72107bfe-f7a3-4d3c-a808-7ebd99e1d809/volumes"
Oct 30 22:05:41.758979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:41.758940    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"]
Oct 30 22:05:41.764404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:41.764379    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:41.782038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:41.782006    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19a725\""
Oct 30 22:05:41.792824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:41.792791    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19a725-dockercfg-4vrvt\""
Oct 30 22:05:41.797019 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:41.796989    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"]
Oct 30 22:05:42.046623 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.043046    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rpm\" (UniqueName: \"kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm\") pod \"payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.046623 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.043212    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.144820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.144775    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rpm\" (UniqueName: \"kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm\") pod \"payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.145014 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.144892    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.145745 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.145722    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.166614 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.166579    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rpm\" (UniqueName: \"kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm\") pod \"payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.381936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.381839    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:42.699091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:42.699054    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"]
Oct 30 22:05:42.701754 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:42.701721    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f91004_2572_4630_a24d_316cf63d8594.slice/crio-d0d529fb17da9f4e820f8f65218d5228680eab6370b10eaf3a9fa401558e28fa WatchSource:0}: Error finding container d0d529fb17da9f4e820f8f65218d5228680eab6370b10eaf3a9fa401558e28fa: Status 404 returned error can't find the container with id d0d529fb17da9f4e820f8f65218d5228680eab6370b10eaf3a9fa401558e28fa
Oct 30 22:05:43.217366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:43.217321    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" event={"ID":"47f91004-2572-4630-a24d-316cf63d8594","Type":"ContainerStarted","Data":"d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085"}
Oct 30 22:05:43.217838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:43.217412    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" event={"ID":"47f91004-2572-4630-a24d-316cf63d8594","Type":"ContainerStarted","Data":"d0d529fb17da9f4e820f8f65218d5228680eab6370b10eaf3a9fa401558e28fa"}
Oct 30 22:05:43.217838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:43.217447    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:43.257467 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:43.257400    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" podStartSLOduration=2.257380035 podStartE2EDuration="2.257380035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:43.251255102 +0000 UTC m=+3326.173713402" watchObservedRunningTime="2026-10-30 22:05:43.257380035 +0000 UTC m=+3326.179838338"
Oct 30 22:05:46.086759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:46.086726    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:05:47.117843 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:47.117808    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:05:47.303157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:47.303118    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"]
Oct 30 22:05:47.309786 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:47.309744    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-db1288-lwxgh"]
Oct 30 22:05:49.140237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:49.140199    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:05:49.140795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:49.140308    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:05:49.223654 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:49.223592    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8831486b-dd73-4155-91eb-8df9e07541d8" path="/var/lib/kubelet/pods/8831486b-dd73-4155-91eb-8df9e07541d8/volumes"
Oct 30 22:05:54.222083 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:54.222049    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:05:55.201927 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.201884    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"]
Oct 30 22:05:55.207527 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.207490    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.226298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.226265    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"]
Oct 30 22:05:55.231664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.231597    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:05:55.231664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.231596    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:05:55.243906 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.243871    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:55.244167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.243878    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:05:55.289492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.289454    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.289750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.289680    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.289845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.289815    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9sf9\" (UniqueName: \"kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.391926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.391886    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.392143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.392026    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sf9\" (UniqueName: \"kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.392143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.392116    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.392934 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.392902    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.394796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.394756    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.438676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.432274    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sf9\" (UniqueName: \"kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.529154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.529116    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:55.863287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:55.863258    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"]
Oct 30 22:05:55.867021 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:05:55.866988    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5670c0_84ab_45d6_a917_d0ddc104d36f.slice/crio-ed601c5654e97b1c478200cd7cf1a6c3e1ab47d538e00867aef0390f4227760c WatchSource:0}: Error finding container ed601c5654e97b1c478200cd7cf1a6c3e1ab47d538e00867aef0390f4227760c: Status 404 returned error can't find the container with id ed601c5654e97b1c478200cd7cf1a6c3e1ab47d538e00867aef0390f4227760c
Oct 30 22:05:56.379290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:56.379245    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg" event={"ID":"7f5670c0-84ab-45d6-a917-d0ddc104d36f","Type":"ContainerStarted","Data":"04b85b1c703d2e49425e687a3ca167ef85a368c502325c2622b28dadeac7a18b"}
Oct 30 22:05:56.379816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:56.379318    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg" event={"ID":"7f5670c0-84ab-45d6-a917-d0ddc104d36f","Type":"ContainerStarted","Data":"ed601c5654e97b1c478200cd7cf1a6c3e1ab47d538e00867aef0390f4227760c"}
Oct 30 22:05:56.420437 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:56.420314    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg" podStartSLOduration=1.42029563 podStartE2EDuration="1.42029563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:56.416524072 +0000 UTC m=+3339.338982382" watchObservedRunningTime="2026-10-30 22:05:56.42029563 +0000 UTC m=+3339.342753951"
Oct 30 22:05:57.390425 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:57.390378    6695 generic.go:356] "Generic (PLEG): container finished" podID="7f5670c0-84ab-45d6-a917-d0ddc104d36f" containerID="04b85b1c703d2e49425e687a3ca167ef85a368c502325c2622b28dadeac7a18b" exitCode=0
Oct 30 22:05:57.390908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:57.390468    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg" event={"ID":"7f5670c0-84ab-45d6-a917-d0ddc104d36f","Type":"ContainerDied","Data":"04b85b1c703d2e49425e687a3ca167ef85a368c502325c2622b28dadeac7a18b"}
Oct 30 22:05:58.596920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.596881    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:58.774050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.774011    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca\") pod \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") "
Oct 30 22:05:58.774284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.774133    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca\") pod \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") "
Oct 30 22:05:58.774284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.774196    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9\" (UniqueName: \"kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9\") pod \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\" (UID: \"7f5670c0-84ab-45d6-a917-d0ddc104d36f\") "
Oct 30 22:05:58.774511 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.774483    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca" pod "7f5670c0-84ab-45d6-a917-d0ddc104d36f" (UID: "7f5670c0-84ab-45d6-a917-d0ddc104d36f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:58.774886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.774868    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:58.776350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.776321    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca" pod "7f5670c0-84ab-45d6-a917-d0ddc104d36f" (UID: "7f5670c0-84ab-45d6-a917-d0ddc104d36f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:58.776479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.776444    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9" pod "7f5670c0-84ab-45d6-a917-d0ddc104d36f" (UID: "7f5670c0-84ab-45d6-a917-d0ddc104d36f"). InnerVolumeSpecName "kube-api-access-h9sf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:58.876909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.876875    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9sf9\" (UniqueName: \"kubernetes.io/projected/7f5670c0-84ab-45d6-a917-d0ddc104d36f-kube-api-access-h9sf9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:58.876909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:58.876909    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5670c0-84ab-45d6-a917-d0ddc104d36f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.037411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:59.037272    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"]
Oct 30 22:05:59.042258 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:59.042215    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d875d3-8mlgq"]
Oct 30 22:05:59.218479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:59.218439    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d36087-dcb8-4cb8-9c99-baebdc35f03a" path="/var/lib/kubelet/pods/15d36087-dcb8-4cb8-9c99-baebdc35f03a/volumes"
Oct 30 22:05:59.410475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:59.410393    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"
Oct 30 22:05:59.410475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:59.410395    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg" event={"ID":"7f5670c0-84ab-45d6-a917-d0ddc104d36f","Type":"ContainerDied","Data":"ed601c5654e97b1c478200cd7cf1a6c3e1ab47d538e00867aef0390f4227760c"}
Oct 30 22:05:59.410475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:05:59.410451    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed601c5654e97b1c478200cd7cf1a6c3e1ab47d538e00867aef0390f4227760c"
Oct 30 22:06:01.084829 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:06:01.084777    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:06:05.689091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:05.689045    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"]
Oct 30 22:06:05.726950 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:05.726909    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"]
Oct 30 22:06:09.534200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:09.534164    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"]
Oct 30 22:06:09.547420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:09.547384    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3172e5-pj5kg"]
Oct 30 22:06:10.689784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:10.689732    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerName="payload-processing" containerID="cri-o://d12d0fed4e6dcf76b6aab04552896421e9ef12e83c94cae044d84e6d2e82aa34" gracePeriod=25
Oct 30 22:06:10.727720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:10.727668    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" podUID="95892cb4-90b8-4996-8162-03da957b2b28" containerName="payload-pre-processing" containerID="cri-o://f0876c04341fd31c4705f7274160c9cddef1dce70fd4d176a31261f9bfcb8d56" gracePeriod=25
Oct 30 22:06:11.217936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:11.217902    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5670c0-84ab-45d6-a917-d0ddc104d36f" path="/var/lib/kubelet/pods/7f5670c0-84ab-45d6-a917-d0ddc104d36f/volumes"
Oct 30 22:06:13.887652 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:13.887593    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:15.084289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:15.084248    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" podUID="95892cb4-90b8-4996-8162-03da957b2b28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:16.450807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:16.450771    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"]
Oct 30 22:06:16.451347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:16.450996    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy" containerID="cri-o://40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6" gracePeriod=30
Oct 30 22:06:17.057366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.057332    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"]
Oct 30 22:06:17.058220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.058203    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7f5670c0-84ab-45d6-a917-d0ddc104d36f" containerName="revoke-keys"
Oct 30 22:06:17.058279 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.058222    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5670c0-84ab-45d6-a917-d0ddc104d36f" containerName="revoke-keys"
Oct 30 22:06:17.058403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.058393    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5670c0-84ab-45d6-a917-d0ddc104d36f" containerName="revoke-keys"
Oct 30 22:06:17.064197 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.064174    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.079332 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.079272    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:17.079332 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.079323    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:17.082848 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.082814    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"]
Oct 30 22:06:17.088031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.087729    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:17.088031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.087885    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:06:17.121223 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.121176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.121447 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.121309    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.121447 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.121411    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd65q\" (UniqueName: \"kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.223183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.223147    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd65q\" (UniqueName: \"kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.223386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.223297    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.223386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.223368    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.223959 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.223936    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.225749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.225729    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.245276 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.245246    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd65q\" (UniqueName: \"kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.380235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.380155    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:17.699361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.699332    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"]
Oct 30 22:06:17.701509 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:06:17.701480    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cbb518_bd3b_4fb6_bb7e_22bfc17a6a1a.slice/crio-e489dacbdcc2781e54a335fc35d6710f54e60fbd36435a6647e9b93cc987c613 WatchSource:0}: Error finding container e489dacbdcc2781e54a335fc35d6710f54e60fbd36435a6647e9b93cc987c613: Status 404 returned error can't find the container with id e489dacbdcc2781e54a335fc35d6710f54e60fbd36435a6647e9b93cc987c613
Oct 30 22:06:17.724360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:17.724324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29" event={"ID":"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a","Type":"ContainerStarted","Data":"e489dacbdcc2781e54a335fc35d6710f54e60fbd36435a6647e9b93cc987c613"}
Oct 30 22:06:18.734301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:18.734268    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29" event={"ID":"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a","Type":"ContainerStarted","Data":"9856eb75f097f945f0019f79ad8bc737b428cb488aecf9d8429937b753072613"}
Oct 30 22:06:18.774668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:18.774579    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29" podStartSLOduration=1.774561262 podStartE2EDuration="1.774561262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:18.773102456 +0000 UTC m=+3361.695560757" watchObservedRunningTime="2026-10-30 22:06:18.774561262 +0000 UTC m=+3361.697019560"
Oct 30 22:06:19.744035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:19.744002    6695 generic.go:356] "Generic (PLEG): container finished" podID="d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" containerID="9856eb75f097f945f0019f79ad8bc737b428cb488aecf9d8429937b753072613" exitCode=0
Oct 30 22:06:19.744529 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:19.744069    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29" event={"ID":"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a","Type":"ContainerDied","Data":"9856eb75f097f945f0019f79ad8bc737b428cb488aecf9d8429937b753072613"}
Oct 30 22:06:20.993546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:20.993502    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:21.116227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.116169    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca\") pod \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") "
Oct 30 22:06:21.116458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.116262    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca\") pod \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") "
Oct 30 22:06:21.116517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.116477    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q\" (UniqueName: \"kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q\") pod \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\" (UID: \"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a\") "
Oct 30 22:06:21.116713 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.116685    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca" pod "d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" (UID: "d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:21.117184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.117164    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.118482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.118460    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca" pod "d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" (UID: "d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:21.118854 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.118831    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q" pod "d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" (UID: "d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a"). InnerVolumeSpecName "kube-api-access-rd65q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:21.218512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.218478    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.218512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.218511    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd65q\" (UniqueName: \"kubernetes.io/projected/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a-kube-api-access-rd65q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.729233 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.729205    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 22:06:21.766059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.766012    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29" event={"ID":"d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a","Type":"ContainerDied","Data":"e489dacbdcc2781e54a335fc35d6710f54e60fbd36435a6647e9b93cc987c613"}
Oct 30 22:06:21.766059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.766046    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"
Oct 30 22:06:21.766349 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.766084    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e489dacbdcc2781e54a335fc35d6710f54e60fbd36435a6647e9b93cc987c613"
Oct 30 22:06:21.768560 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.768530    6695 generic.go:356] "Generic (PLEG): container finished" podID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerID="40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6" exitCode=0
Oct 30 22:06:21.768751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.768628    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"
Oct 30 22:06:21.768751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.768701    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" event={"ID":"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73","Type":"ContainerDied","Data":"40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6"}
Oct 30 22:06:21.768884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.768770    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp" event={"ID":"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73","Type":"ContainerDied","Data":"10b06ae035810c809d544c2e2a105a7cb25709d1b2c3603cabb61b4423f9a453"}
Oct 30 22:06:21.768884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.768796    6695 scope.go:122] "RemoveContainer" containerID="40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6"
Oct 30 22:06:21.781323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.781300    6695 scope.go:122] "RemoveContainer" containerID="10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"
Oct 30 22:06:21.794823 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.794785    6695 scope.go:122] "RemoveContainer" containerID="40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6"
Oct 30 22:06:21.795202 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:06:21.795177    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6\": container with ID starting with 40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6 not found: ID does not exist" containerID="40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6"
Oct 30 22:06:21.795314 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.795212    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6"} err="failed to get container status \"40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6\": rpc error: code = NotFound desc = could not find container \"40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6\": container with ID starting with 40f4c0f33bae80dc74f4c10709231e7eae9fb1922c21560a37d88c320b1e16e6 not found: ID does not exist"
Oct 30 22:06:21.795314 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.795235    6695 scope.go:122] "RemoveContainer" containerID="10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"
Oct 30 22:06:21.795564 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:06:21.795538    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0\": container with ID starting with 10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0 not found: ID does not exist" containerID="10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"
Oct 30 22:06:21.795695 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.795571    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0"} err="failed to get container status \"10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0\": rpc error: code = NotFound desc = could not find container \"10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0\": container with ID starting with 10001c289c216cf74aa421f4822f0aed73644582b88d70946020339e558bbbb0 not found: ID does not exist"
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828181    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828294    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828359    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828394    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828491    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828570    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.828596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828629    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.829182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.828916    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.829182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.829005    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert\") pod \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\" (UID: \"ee71de1e-1b94-48d9-a8c6-f7856bb6ea73\") "
Oct 30 22:06:21.830251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.830184    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:21.830738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.830701    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:21.830941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.830915    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:21.831035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.830927    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:21.833493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.833459    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:21.835594 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.835562    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:21.838376 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.838348    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "kube-api-access-qg88q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:21.840008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.839984    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:21.841240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.841210    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo" pod "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" (UID: "ee71de1e-1b94-48d9-a8c6-f7856bb6ea73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931052    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931083    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931093    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg88q\" (UniqueName: \"kubernetes.io/projected/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-kube-api-access-qg88q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931105    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931115    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931123    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931132    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931143    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:21.931209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:21.931151    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:22.118107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:22.118063    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"]
Oct 30 22:06:22.122777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:22.122743    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3172e5-openshift-default-5c8c5f4cf7-s5skp"]
Oct 30 22:06:23.217317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:23.217278    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" path="/var/lib/kubelet/pods/ee71de1e-1b94-48d9-a8c6-f7856bb6ea73/volumes"
Oct 30 22:06:23.888482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:23.888439    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:25.084456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:25.084413    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" podUID="95892cb4-90b8-4996-8162-03da957b2b28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:27.557132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:27.557082    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"]
Oct 30 22:06:28.891563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:28.891522    6695 scope.go:122] "RemoveContainer" containerID="e04880d14c71a0ef54e6373b1320011d7ed5dc1faedaac601403e148b9af83c4"
Oct 30 22:06:28.904686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:28.904652    6695 scope.go:122] "RemoveContainer" containerID="a6310bb66a1c4ee194b615fdffa95c5b97e1f7071aaef93ba9c9f56570830138"
Oct 30 22:06:30.900354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:30.900314    6695 generic.go:356] "Generic (PLEG): container finished" podID="95892cb4-90b8-4996-8162-03da957b2b28" containerID="f0876c04341fd31c4705f7274160c9cddef1dce70fd4d176a31261f9bfcb8d56" exitCode=0
Oct 30 22:06:30.900877 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:30.900492    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" event={"ID":"95892cb4-90b8-4996-8162-03da957b2b28","Type":"ContainerDied","Data":"f0876c04341fd31c4705f7274160c9cddef1dce70fd4d176a31261f9bfcb8d56"}
Oct 30 22:06:30.902847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:30.902815    6695 generic.go:356] "Generic (PLEG): container finished" podID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerID="d12d0fed4e6dcf76b6aab04552896421e9ef12e83c94cae044d84e6d2e82aa34" exitCode=0
Oct 30 22:06:30.903113 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:30.903092    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" event={"ID":"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6","Type":"ContainerDied","Data":"d12d0fed4e6dcf76b6aab04552896421e9ef12e83c94cae044d84e6d2e82aa34"}
Oct 30 22:06:31.048070 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.048029    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 22:06:31.085528 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:06:31.085482    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:06:31.092474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.092443    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 22:06:31.157371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.157333    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume\") pod \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") "
Oct 30 22:06:31.157735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.157707    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc\" (UniqueName: \"kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc\") pod \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\" (UID: \"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6\") "
Oct 30 22:06:31.158218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.158190    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume" pod "84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" (UID: "84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:31.158465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.158448    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:31.160969 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.160938    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc" pod "84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" (UID: "84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6"). InnerVolumeSpecName "kube-api-access-qj6sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:31.259732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.259631    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676\" (UniqueName: \"kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676\") pod \"95892cb4-90b8-4996-8162-03da957b2b28\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") "
Oct 30 22:06:31.259732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.259680    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume\") pod \"95892cb4-90b8-4996-8162-03da957b2b28\" (UID: \"95892cb4-90b8-4996-8162-03da957b2b28\") "
Oct 30 22:06:31.260106 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.260081    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume" pod "95892cb4-90b8-4996-8162-03da957b2b28" (UID: "95892cb4-90b8-4996-8162-03da957b2b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:31.260348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.260334    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95892cb4-90b8-4996-8162-03da957b2b28-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:31.260400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.260351    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6sc\" (UniqueName: \"kubernetes.io/projected/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6-kube-api-access-qj6sc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:31.261942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.261914    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676" pod "95892cb4-90b8-4996-8162-03da957b2b28" (UID: "95892cb4-90b8-4996-8162-03da957b2b28"). InnerVolumeSpecName "kube-api-access-lr676". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:31.361990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.361952    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr676\" (UniqueName: \"kubernetes.io/projected/95892cb4-90b8-4996-8162-03da957b2b28-kube-api-access-lr676\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:31.919339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.919305    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"
Oct 30 22:06:31.919875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.919345    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t" event={"ID":"84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6","Type":"ContainerDied","Data":"512b322da04b7c4024d481bd565a4616575ebac0d042ca47c087b2fd9eb0c3db"}
Oct 30 22:06:31.919875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.919423    6695 scope.go:122] "RemoveContainer" containerID="d12d0fed4e6dcf76b6aab04552896421e9ef12e83c94cae044d84e6d2e82aa34"
Oct 30 22:06:31.923075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.922717    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf" event={"ID":"95892cb4-90b8-4996-8162-03da957b2b28","Type":"ContainerDied","Data":"14df8933c8adc4e53305fd2f5b954db5a65d7aa4c828ed7d3449a02b6696bcbc"}
Oct 30 22:06:31.923075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.922850    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"
Oct 30 22:06:31.941958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.941921    6695 scope.go:122] "RemoveContainer" containerID="f0876c04341fd31c4705f7274160c9cddef1dce70fd4d176a31261f9bfcb8d56"
Oct 30 22:06:31.985899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.985787    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"]
Oct 30 22:06:31.996498 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:31.996464    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3172e5-6f8645b479-qw72t"]
Oct 30 22:06:32.020234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:32.020194    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"]
Oct 30 22:06:32.025300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:32.025256    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3172e5-74467ccd95-sgchf"]
Oct 30 22:06:32.557789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:32.557739    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerName="payload-processing" containerID="cri-o://bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a" gracePeriod=25
Oct 30 22:06:32.756799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:32.756751    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"]
Oct 30 22:06:32.763151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:32.763107    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f49db2-jnt29"]
Oct 30 22:06:33.218502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:33.218458    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" path="/var/lib/kubelet/pods/84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6/volumes"
Oct 30 22:06:33.219045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:33.219029    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95892cb4-90b8-4996-8162-03da957b2b28" path="/var/lib/kubelet/pods/95892cb4-90b8-4996-8162-03da957b2b28/volumes"
Oct 30 22:06:33.219474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:33.219459    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" path="/var/lib/kubelet/pods/d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a/volumes"
Oct 30 22:06:38.884787 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:38.884743    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:40.550994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.550951    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"]
Oct 30 22:06:40.551875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551856    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" containerName="revoke-keys"
Oct 30 22:06:40.551875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551875    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" containerName="revoke-keys"
Oct 30 22:06:40.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551907    6695 cpu_manager.go:414] "containerMap: removing container" podUID="95892cb4-90b8-4996-8162-03da957b2b28" containerName="payload-pre-processing"
Oct 30 22:06:40.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551913    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95892cb4-90b8-4996-8162-03da957b2b28" containerName="payload-pre-processing"
Oct 30 22:06:40.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551938    6695 cpu_manager.go:414] "containerMap: removing container" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerName="payload-processing"
Oct 30 22:06:40.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551943    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerName="payload-processing"
Oct 30 22:06:40.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551954    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy"
Oct 30 22:06:40.551970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551959    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551977    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.551982    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.552098    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cbb518-bd3b-4fb6-bb7e-22bfc17a6a1a" containerName="revoke-keys"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.552113    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="95892cb4-90b8-4996-8162-03da957b2b28" containerName="payload-pre-processing"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.552123    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bac7c3-83b1-4ad1-85b6-f16e59a8b8d6" containerName="payload-processing"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.552129    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy"
Oct 30 22:06:40.552170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.552138    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee71de1e-1b94-48d9-a8c6-f7856bb6ea73" containerName="istio-proxy"
Oct 30 22:06:40.557972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.557942    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.576041 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.576000    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"]
Oct 30 22:06:40.580558 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.580524    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:40.580769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.580561    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:40.580769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.580646    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:40.583340 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.583313    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:06:40.616029 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.615994    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.616251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.616043    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mp24\" (UniqueName: \"kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.616251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.616079    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.717932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.717887    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.718132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.717946    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mp24\" (UniqueName: \"kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.718132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.717980    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.718778 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.718751    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.720805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.720770    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.738334 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.738302    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mp24\" (UniqueName: \"kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24\") pod \"maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:40.876681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:40.876540    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:41.200107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:41.200071    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"]
Oct 30 22:06:41.202786 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:06:41.202753    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33719f3a_c049_49c5_bc1e_4d0114d6c1e1.slice/crio-7ba60fb5f949b99529fddef4664df5c332fa69cb3427d2520a24e040f150be1b WatchSource:0}: Error finding container 7ba60fb5f949b99529fddef4664df5c332fa69cb3427d2520a24e040f150be1b: Status 404 returned error can't find the container with id 7ba60fb5f949b99529fddef4664df5c332fa69cb3427d2520a24e040f150be1b
Oct 30 22:06:42.087303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:42.087262    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd" event={"ID":"33719f3a-c049-49c5-bc1e-4d0114d6c1e1","Type":"ContainerStarted","Data":"e7db03b5eb13321d2975686c7acdd0f1a9426fb3112bafe1a873e70056df7360"}
Oct 30 22:06:42.087865 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:42.087324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd" event={"ID":"33719f3a-c049-49c5-bc1e-4d0114d6c1e1","Type":"ContainerStarted","Data":"7ba60fb5f949b99529fddef4664df5c332fa69cb3427d2520a24e040f150be1b"}
Oct 30 22:06:42.126194 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:42.126123    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd" podStartSLOduration=2.126104688 podStartE2EDuration="2.126104688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:42.124860968 +0000 UTC m=+3385.047319248" watchObservedRunningTime="2026-10-30 22:06:42.126104688 +0000 UTC m=+3385.048562987"
Oct 30 22:06:43.098175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:43.098140    6695 generic.go:356] "Generic (PLEG): container finished" podID="33719f3a-c049-49c5-bc1e-4d0114d6c1e1" containerID="e7db03b5eb13321d2975686c7acdd0f1a9426fb3112bafe1a873e70056df7360" exitCode=0
Oct 30 22:06:43.098668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:43.098201    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd" event={"ID":"33719f3a-c049-49c5-bc1e-4d0114d6c1e1","Type":"ContainerDied","Data":"e7db03b5eb13321d2975686c7acdd0f1a9426fb3112bafe1a873e70056df7360"}
Oct 30 22:06:44.290210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.290169    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:44.380480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.380357    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca\") pod \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") "
Oct 30 22:06:44.380729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.380487    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca\") pod \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") "
Oct 30 22:06:44.380814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.380724    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24\" (UniqueName: \"kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24\") pod \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\" (UID: \"33719f3a-c049-49c5-bc1e-4d0114d6c1e1\") "
Oct 30 22:06:44.380891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.380847    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca" pod "33719f3a-c049-49c5-bc1e-4d0114d6c1e1" (UID: "33719f3a-c049-49c5-bc1e-4d0114d6c1e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:44.382087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.381800    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:44.383238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.383211    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca" pod "33719f3a-c049-49c5-bc1e-4d0114d6c1e1" (UID: "33719f3a-c049-49c5-bc1e-4d0114d6c1e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:44.383398 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.383275    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24" pod "33719f3a-c049-49c5-bc1e-4d0114d6c1e1" (UID: "33719f3a-c049-49c5-bc1e-4d0114d6c1e1"). InnerVolumeSpecName "kube-api-access-4mp24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:44.484035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.483979    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:44.484035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:44.484023    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mp24\" (UniqueName: \"kubernetes.io/projected/33719f3a-c049-49c5-bc1e-4d0114d6c1e1-kube-api-access-4mp24\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:45.116254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:45.116217    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"
Oct 30 22:06:45.116254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:45.116246    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd" event={"ID":"33719f3a-c049-49c5-bc1e-4d0114d6c1e1","Type":"ContainerDied","Data":"7ba60fb5f949b99529fddef4664df5c332fa69cb3427d2520a24e040f150be1b"}
Oct 30 22:06:45.116496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:45.116299    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba60fb5f949b99529fddef4664df5c332fa69cb3427d2520a24e040f150be1b"
Oct 30 22:06:45.280962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:45.280922    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"]
Oct 30 22:06:45.300341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:45.300304    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"]
Oct 30 22:06:48.885525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:48.885489    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:51.033836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:51.033789    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"]
Oct 30 22:06:51.065947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:51.065906    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"]
Oct 30 22:06:52.888413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:52.888382    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 22:06:52.961495 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:52.961410    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume\") pod \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") "
Oct 30 22:06:52.962082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:52.961900    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume" pod "2e70a689-84d1-4aee-84c9-fe40c8b801d4" (UID: "2e70a689-84d1-4aee-84c9-fe40c8b801d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:52.962280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:52.962119    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98\" (UniqueName: \"kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98\") pod \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\" (UID: \"2e70a689-84d1-4aee-84c9-fe40c8b801d4\") "
Oct 30 22:06:52.963240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:52.963214    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e70a689-84d1-4aee-84c9-fe40c8b801d4-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:52.964953 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:52.964913    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98" pod "2e70a689-84d1-4aee-84c9-fe40c8b801d4" (UID: "2e70a689-84d1-4aee-84c9-fe40c8b801d4"). InnerVolumeSpecName "kube-api-access-zqh98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:53.065620 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.065564    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqh98\" (UniqueName: \"kubernetes.io/projected/2e70a689-84d1-4aee-84c9-fe40c8b801d4-kube-api-access-zqh98\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:06:53.230449 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.230364    6695 generic.go:356] "Generic (PLEG): container finished" podID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerID="bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a" exitCode=0
Oct 30 22:06:53.230449 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.230436    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"
Oct 30 22:06:53.230756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.230454    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" event={"ID":"2e70a689-84d1-4aee-84c9-fe40c8b801d4","Type":"ContainerDied","Data":"bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a"}
Oct 30 22:06:53.230756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.230516    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9" event={"ID":"2e70a689-84d1-4aee-84c9-fe40c8b801d4","Type":"ContainerDied","Data":"e802d86d185aee4512a3805afe33b581395f32633002072eb56905fce03510d3"}
Oct 30 22:06:53.230756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.230545    6695 scope.go:122] "RemoveContainer" containerID="bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a"
Oct 30 22:06:53.245091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.245064    6695 scope.go:122] "RemoveContainer" containerID="bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a"
Oct 30 22:06:53.245479 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:06:53.245451    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a\": container with ID starting with bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a not found: ID does not exist" containerID="bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a"
Oct 30 22:06:53.245574 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.245492    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a"} err="failed to get container status \"bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a\": rpc error: code = NotFound desc = could not find container \"bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a\": container with ID starting with bafd1c4cfc158b3fe946f7f19d45dde3d2d6cbe6b0cb7d1abd3d14369d59ee7a not found: ID does not exist"
Oct 30 22:06:53.284147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.284105    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"]
Oct 30 22:06:53.295287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:53.295244    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f49db2-5488cd49f7-wrqr9"]
Oct 30 22:06:55.218267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:55.218235    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" path="/var/lib/kubelet/pods/2e70a689-84d1-4aee-84c9-fe40c8b801d4/volumes"
Oct 30 22:06:56.035836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:56.035780    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing" containerID="cri-o://dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551" gracePeriod=25
Oct 30 22:06:56.067195 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:56.067148    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing" containerID="cri-o://919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261" gracePeriod=25
Oct 30 22:06:56.336546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:56.336330    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:56.336546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:06:56.336388    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:07:00.310835 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:00.310784    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" containerID="cri-o://f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c" gracePeriod=45
Oct 30 22:07:00.342048 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:00.341983    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" containerID="cri-o://e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b" gracePeriod=45
Oct 30 22:07:00.675633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:00.669918    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused"
Oct 30 22:07:00.986293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:00.986260    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 22:07:01.067034 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.066551    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 22:07:01.089008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.088961    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache\") pod \"75b95c9e-4fd2-402b-ad68-0025760c7394\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") "
Oct 30 22:07:01.089285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.089085    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j\" (UniqueName: \"kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j\") pod \"75b95c9e-4fd2-402b-ad68-0025760c7394\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") "
Oct 30 22:07:01.089285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.089146    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs\" (UniqueName: \"kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs\") pod \"75b95c9e-4fd2-402b-ad68-0025760c7394\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") "
Oct 30 22:07:01.089285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.089181    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir\") pod \"75b95c9e-4fd2-402b-ad68-0025760c7394\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") "
Oct 30 22:07:01.089569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.089388    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home\") pod \"75b95c9e-4fd2-402b-ad68-0025760c7394\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") "
Oct 30 22:07:01.089569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.089484    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm\") pod \"75b95c9e-4fd2-402b-ad68-0025760c7394\" (UID: \"75b95c9e-4fd2-402b-ad68-0025760c7394\") "
Oct 30 22:07:01.091884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.091823    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache" pod "75b95c9e-4fd2-402b-ad68-0025760c7394" (UID: "75b95c9e-4fd2-402b-ad68-0025760c7394"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.092149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.092110    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home" pod "75b95c9e-4fd2-402b-ad68-0025760c7394" (UID: "75b95c9e-4fd2-402b-ad68-0025760c7394"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.092460 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.092402    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir" pod "75b95c9e-4fd2-402b-ad68-0025760c7394" (UID: "75b95c9e-4fd2-402b-ad68-0025760c7394"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.093404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.093363    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm" pod "75b95c9e-4fd2-402b-ad68-0025760c7394" (UID: "75b95c9e-4fd2-402b-ad68-0025760c7394"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.093961 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.093921    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j" pod "75b95c9e-4fd2-402b-ad68-0025760c7394" (UID: "75b95c9e-4fd2-402b-ad68-0025760c7394"). InnerVolumeSpecName "kube-api-access-zxv8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:01.098066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.098026    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs" pod "75b95c9e-4fd2-402b-ad68-0025760c7394" (UID: "75b95c9e-4fd2-402b-ad68-0025760c7394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:01.114670 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:07:01.114628    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.191890    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs\") pod \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") "
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.192108    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home\") pod \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") "
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.192248    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw\" (UniqueName: \"kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw\") pod \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") "
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.192368    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm\") pod \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") "
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.192407    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir\") pod \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") "
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.192489    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache\") pod \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\" (UID: \"43fcd1a9-8a47-47c9-8a06-870c1dec14b6\") "
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.193320    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.193352    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxv8j\" (UniqueName: \"kubernetes.io/projected/75b95c9e-4fd2-402b-ad68-0025760c7394-kube-api-access-zxv8j\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.193369    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b95c9e-4fd2-402b-ad68-0025760c7394-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.193383    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.193396    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.193705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.193409    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b95c9e-4fd2-402b-ad68-0025760c7394-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.194676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.194587    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home" pod "43fcd1a9-8a47-47c9-8a06-870c1dec14b6" (UID: "43fcd1a9-8a47-47c9-8a06-870c1dec14b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.194857 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.194707    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache" pod "43fcd1a9-8a47-47c9-8a06-870c1dec14b6" (UID: "43fcd1a9-8a47-47c9-8a06-870c1dec14b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.194927 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.194848    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir" pod "43fcd1a9-8a47-47c9-8a06-870c1dec14b6" (UID: "43fcd1a9-8a47-47c9-8a06-870c1dec14b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.197768 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.197708    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm" pod "43fcd1a9-8a47-47c9-8a06-870c1dec14b6" (UID: "43fcd1a9-8a47-47c9-8a06-870c1dec14b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:01.198785 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.198731    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs" pod "43fcd1a9-8a47-47c9-8a06-870c1dec14b6" (UID: "43fcd1a9-8a47-47c9-8a06-870c1dec14b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:01.200597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.200515    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw" pod "43fcd1a9-8a47-47c9-8a06-870c1dec14b6" (UID: "43fcd1a9-8a47-47c9-8a06-870c1dec14b6"). InnerVolumeSpecName "kube-api-access-jnxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:01.301066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.298886    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.301066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.298970    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.301066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.298985    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.301066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.299000    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.301066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.299012    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.301066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.299030    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxjw\" (UniqueName: \"kubernetes.io/projected/43fcd1a9-8a47-47c9-8a06-870c1dec14b6-kube-api-access-jnxjw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.365008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.364965    6695 generic.go:356] "Generic (PLEG): container finished" podID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerID="e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b" exitCode=0
Oct 30 22:07:01.365457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.365033    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"
Oct 30 22:07:01.365457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.365061    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" event={"ID":"43fcd1a9-8a47-47c9-8a06-870c1dec14b6","Type":"ContainerDied","Data":"e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b"}
Oct 30 22:07:01.365457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.365101    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z" event={"ID":"43fcd1a9-8a47-47c9-8a06-870c1dec14b6","Type":"ContainerDied","Data":"8b885d6e31cd74980ae8bc4130c230d7110d80f3980a064568ce1761207ccb3b"}
Oct 30 22:07:01.365457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.365119    6695 scope.go:122] "RemoveContainer" containerID="e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b"
Oct 30 22:07:01.367219 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.367188    6695 generic.go:356] "Generic (PLEG): container finished" podID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerID="f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c" exitCode=0
Oct 30 22:07:01.367368 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.367277    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"
Oct 30 22:07:01.367368 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.367344    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" event={"ID":"75b95c9e-4fd2-402b-ad68-0025760c7394","Type":"ContainerDied","Data":"f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c"}
Oct 30 22:07:01.367489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.367383    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" event={"ID":"75b95c9e-4fd2-402b-ad68-0025760c7394","Type":"ContainerDied","Data":"dd5f0c0bb23b6da563c2e981bd485c8715b7f147471891a1be8ae9359b261d99"}
Oct 30 22:07:01.381866 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.381835    6695 scope.go:122] "RemoveContainer" containerID="e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b"
Oct 30 22:07:01.382437 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:07:01.382377    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b\": container with ID starting with e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b not found: ID does not exist" containerID="e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b"
Oct 30 22:07:01.382557 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.382462    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b"} err="failed to get container status \"e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b\": rpc error: code = NotFound desc = could not find container \"e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b\": container with ID starting with e9356cf233d6535b7a120840fc3d8878033e0a16d7c4606f7cbac148ca85866b not found: ID does not exist"
Oct 30 22:07:01.382557 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.382503    6695 scope.go:122] "RemoveContainer" containerID="f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c"
Oct 30 22:07:01.440833 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.440793    6695 scope.go:122] "RemoveContainer" containerID="f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c"
Oct 30 22:07:01.441563 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:07:01.441534    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c\": container with ID starting with f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c not found: ID does not exist" containerID="f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c"
Oct 30 22:07:01.441728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.441571    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c"} err="failed to get container status \"f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c\": rpc error: code = NotFound desc = could not find container \"f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c\": container with ID starting with f4370de89cfd4b344f41ec2ea93819219dd3148187f1d158c2231ea80fc7fd5c not found: ID does not exist"
Oct 30 22:07:01.443798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.443777    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"]
Oct 30 22:07:01.447518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.447481    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq"]
Oct 30 22:07:01.474982 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.474892    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"]
Oct 30 22:07:01.479655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.479617    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/simulator-w0-49738b-kserve-96d44c498-nd49z"]
Oct 30 22:07:01.710067 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:01.710018    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/distinct-w0-49738b-kserve-56d4667b9d-zv7wq" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:03.240330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:03.240285    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" path="/var/lib/kubelet/pods/43fcd1a9-8a47-47c9-8a06-870c1dec14b6/volumes"
Oct 30 22:07:03.241033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:03.241014    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" path="/var/lib/kubelet/pods/75b95c9e-4fd2-402b-ad68-0025760c7394/volumes"
Oct 30 22:07:04.005730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.005679    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:07:04.006071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.006040    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:04.890432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.890379    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"]
Oct 30 22:07:04.891613 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891567    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerName="payload-processing"
Oct 30 22:07:04.891613 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891614    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerName="payload-processing"
Oct 30 22:07:04.891822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891683    6695 cpu_manager.go:414] "containerMap: removing container" podUID="33719f3a-c049-49c5-bc1e-4d0114d6c1e1" containerName="revoke-keys"
Oct 30 22:07:04.891822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891693    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33719f3a-c049-49c5-bc1e-4d0114d6c1e1" containerName="revoke-keys"
Oct 30 22:07:04.891822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891709    6695 cpu_manager.go:414] "containerMap: removing container" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main"
Oct 30 22:07:04.891822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891717    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main"
Oct 30 22:07:04.891822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891759    6695 cpu_manager.go:414] "containerMap: removing container" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main"
Oct 30 22:07:04.891822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.891766    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main"
Oct 30 22:07:04.892182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.892064    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fcd1a9-8a47-47c9-8a06-870c1dec14b6" containerName="main"
Oct 30 22:07:04.892182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.892087    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e70a689-84d1-4aee-84c9-fe40c8b801d4" containerName="payload-processing"
Oct 30 22:07:04.892182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.892110    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="33719f3a-c049-49c5-bc1e-4d0114d6c1e1" containerName="revoke-keys"
Oct 30 22:07:04.892182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.892122    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b95c9e-4fd2-402b-ad68-0025760c7394" containerName="main"
Oct 30 22:07:04.948626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.945228    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:04.955426 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.955389    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"]
Oct 30 22:07:04.978082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.978043    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d53e0\"/\"test-model-8d53e0-kserve-self-signed-certs\""
Oct 30 22:07:04.978380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.978163    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d53e0\"/\"kube-root-ca.crt\""
Oct 30 22:07:04.978380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.978222    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d53e0\"/\"openshift-service-ca.crt\""
Oct 30 22:07:04.978380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.978223    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d53e0\"/\"default-dockercfg-n28vb\""
Oct 30 22:07:04.995100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.995055    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:04.995302 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.995277    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:04.995374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.995326    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:04.995550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.995524    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7ms\" (UniqueName: \"kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:04.995628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.995588    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:04.995769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:04.995753    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.097962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.097916    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.097980    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098092    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7ms\" (UniqueName: \"kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098512    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098565    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098635    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098720    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.098877 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098859    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.099012 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.098994    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.268187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.267017    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.345234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.316325    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7ms\" (UniqueName: \"kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.351632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.351338    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs\") pod \"test-model-8d53e0-kserve-7d9dbbc765-xmzq4\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") " pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:05.355735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:05.355624    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:06.317208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:06.317160    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:07:06.318251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:06.318212    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:07:06.562580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:06.562532    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"]
Oct 30 22:07:06.566409 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:06.565658    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdeddcd0_d866_4ebc_867b_f5184af92bb1.slice/crio-0f424ccc7bce6a2a917dfcef46d3f6c95488ebd78d56cf77d072e43d2abe0f2c WatchSource:0}: Error finding container 0f424ccc7bce6a2a917dfcef46d3f6c95488ebd78d56cf77d072e43d2abe0f2c: Status 404 returned error can't find the container with id 0f424ccc7bce6a2a917dfcef46d3f6c95488ebd78d56cf77d072e43d2abe0f2c
Oct 30 22:07:07.261757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.261702    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"]
Oct 30 22:07:07.269718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.269685    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.300032 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.295682    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-589ebc-openshift-default-dockercfg-ldlwx\""
Oct 30 22:07:07.307596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.307556    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"]
Oct 30 22:07:07.381583 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.381537    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.381614    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.381765    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.381806    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.381941    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.382099    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5km8\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.382130    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.382159    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.382263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.382180    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.491903    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.491973    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492035    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492071    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492134    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492258    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5km8\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492292    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492327    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.495669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.492354    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.512419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.509323    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.512419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.510053    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.514531 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.512983    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.514531 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.513662    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.515753 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.514975    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.521238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.519111    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" event={"ID":"bdeddcd0-d866-4ebc-867b-f5184af92bb1","Type":"ContainerStarted","Data":"b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb"}
Oct 30 22:07:07.521238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.519169    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" event={"ID":"bdeddcd0-d866-4ebc-867b-f5184af92bb1","Type":"ContainerStarted","Data":"0f424ccc7bce6a2a917dfcef46d3f6c95488ebd78d56cf77d072e43d2abe0f2c"}
Oct 30 22:07:07.577389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.574075    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.577389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.576209    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.593750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.585562    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5km8\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.620918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.599540    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token\") pod \"e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") " pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:07.620918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:07.609932    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:08.149056 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.148980    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" podStartSLOduration=4.14895329 podStartE2EDuration="4.14895329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:07.639906695 +0000 UTC m=+3410.562365001" watchObservedRunningTime="2026-10-30 22:07:08.14895329 +0000 UTC m=+3411.071411588"
Oct 30 22:07:08.155348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.155306    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"]
Oct 30 22:07:08.158189 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:08.158151    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe820873_6dc6_44a7_8813_6902c183c856.slice/crio-1f10b6bf91a0cb04f9b799fdb5a12c383f8b1999f567306b5a76fa1f45146d3c WatchSource:0}: Error finding container 1f10b6bf91a0cb04f9b799fdb5a12c383f8b1999f567306b5a76fa1f45146d3c: Status 404 returned error can't find the container with id 1f10b6bf91a0cb04f9b799fdb5a12c383f8b1999f567306b5a76fa1f45146d3c
Oct 30 22:07:08.162169 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.162124    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:08.162323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.162222    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:08.162323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.162269    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:08.357289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.356195    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:08.536793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.536737    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" event={"ID":"fe820873-6dc6-44a7-8813-6902c183c856","Type":"ContainerStarted","Data":"304cf1235d6d2691f6e379c9f4b81120e0fcb7c6aa78b0487bcea5f42a910712"}
Oct 30 22:07:08.537404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.536826    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" event={"ID":"fe820873-6dc6-44a7-8813-6902c183c856","Type":"ContainerStarted","Data":"1f10b6bf91a0cb04f9b799fdb5a12c383f8b1999f567306b5a76fa1f45146d3c"}
Oct 30 22:07:08.610991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:08.610845    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:09.619179 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:09.619134    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:09.687489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:09.687416    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podStartSLOduration=2.687394743 podStartE2EDuration="2.687394743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:08.586274872 +0000 UTC m=+3411.508733171" watchObservedRunningTime="2026-10-30 22:07:09.687394743 +0000 UTC m=+3412.609853042"
Oct 30 22:07:10.574448 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:10.574384    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:10.582147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:10.581999    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:07:12.605708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.605669    6695 generic.go:356] "Generic (PLEG): container finished" podID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerID="2e32d1804e8470755d00c3996eb58f19cbe8e8e57a4de5808503c5974fb8e3f5" exitCode=0
Oct 30 22:07:12.606257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.605748    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" event={"ID":"3c2f7991-55c5-4109-b142-d8371f73f09c","Type":"ContainerDied","Data":"2e32d1804e8470755d00c3996eb58f19cbe8e8e57a4de5808503c5974fb8e3f5"}
Oct 30 22:07:12.606355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.606334    6695 scope.go:122] "RemoveContainer" containerID="2e32d1804e8470755d00c3996eb58f19cbe8e8e57a4de5808503c5974fb8e3f5"
Oct 30 22:07:12.607996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.607968    6695 generic.go:356] "Generic (PLEG): container finished" podID="79a33640-8558-4199-b50c-2c64140f338a" containerID="7896cbd2b09f2af560736ccc12171e0ed7c021e02a97c512993ff0badf5d87dc" exitCode=0
Oct 30 22:07:12.608138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.608037    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" event={"ID":"79a33640-8558-4199-b50c-2c64140f338a","Type":"ContainerDied","Data":"7896cbd2b09f2af560736ccc12171e0ed7c021e02a97c512993ff0badf5d87dc"}
Oct 30 22:07:12.608487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.608471    6695 scope.go:122] "RemoveContainer" containerID="7896cbd2b09f2af560736ccc12171e0ed7c021e02a97c512993ff0badf5d87dc"
Oct 30 22:07:12.608579 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.608546    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:12.608757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.608735    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:12.608856 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.608797    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:12.610310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.610281    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:12.610405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.610347    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:12.610405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:12.610379    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:13.672758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.672722    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"]
Oct 30 22:07:13.678707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.678676    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" event={"ID":"3c2f7991-55c5-4109-b142-d8371f73f09c","Type":"ContainerStarted","Data":"7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550"}
Oct 30 22:07:13.678940 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.678791    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:13.680450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.680425    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" event={"ID":"79a33640-8558-4199-b50c-2c64140f338a","Type":"ContainerStarted","Data":"de7070be8f8d80cd73db8b00d0dc6c76b0f7325d2a1f8f5dacfd7a521c3558f2"}
Oct 30 22:07:13.694286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.694252    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"]
Oct 30 22:07:13.699792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.699766    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-589ebc-dockercfg-w46t8\""
Oct 30 22:07:13.704669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.704591    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-589ebc\""
Oct 30 22:07:13.801907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.791676    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:07:13.801907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.791714    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused" start-of-body=
Oct 30 22:07:13.801907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.791751    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:07:13.801907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.791767    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused"
Oct 30 22:07:13.851631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.794029    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:13.851631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.827051    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhdx\" (UniqueName: \"kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx\") pod \"payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:13.851631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.820916    6695 patch_prober.go:29] interesting pod/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused" start-of-body=
Oct 30 22:07:13.870626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.864081    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused"
Oct 30 22:07:13.961746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.958287    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:13.961746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.958389    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhdx\" (UniqueName: \"kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx\") pod \"payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:13.961746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:13.960131    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:14.098429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.098386    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"]
Oct 30 22:07:14.106284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.106240    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.162628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.161020    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhdx\" (UniqueName: \"kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx\") pod \"payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:14.169640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.168824    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"]
Oct 30 22:07:14.259937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.257380    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.259937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.257648    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whlbn\" (UniqueName: \"kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn\") pod \"payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.297499 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.297458    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:14.376235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.376185    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.376444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.376406    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whlbn\" (UniqueName: \"kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn\") pod \"payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.378346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.378128    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.399277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.398670    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whlbn\" (UniqueName: \"kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn\") pod \"payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.483026 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.482985    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:14.775834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.767736    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused" start-of-body=
Oct 30 22:07:14.775834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.767805    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused"
Oct 30 22:07:14.775834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.768506    6695 patch_prober.go:29] interesting pod/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused" start-of-body=
Oct 30 22:07:14.775834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.768552    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused"
Oct 30 22:07:14.775834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.774101    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"]
Oct 30 22:07:14.776972 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:14.776935    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69963d5e_752a_4fe1_835e_23f0a83a58a3.slice/crio-1af720e4f52835769d61b77c6873ff24954072f52c07fd926e537e802094b310 WatchSource:0}: Error finding container 1af720e4f52835769d61b77c6873ff24954072f52c07fd926e537e802094b310: Status 404 returned error can't find the container with id 1af720e4f52835769d61b77c6873ff24954072f52c07fd926e537e802094b310
Oct 30 22:07:14.876626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:14.876048    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:07:15.005629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.005317    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"]
Oct 30 22:07:15.361631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.358892    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:15.376896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.376865    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:15.706919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.706878    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht" event={"ID":"58568bf1-662b-4a1c-aac6-e43ea30616cf","Type":"ContainerStarted","Data":"453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1"}
Oct 30 22:07:15.707154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.706933    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht" event={"ID":"58568bf1-662b-4a1c-aac6-e43ea30616cf","Type":"ContainerStarted","Data":"f74b664e1eb45a15a587dee56602dbffa97fbbee97d66d9701001c485fc35b46"}
Oct 30 22:07:15.707154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.706977    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:15.709530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.709494    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" event={"ID":"69963d5e-752a-4fe1-835e-23f0a83a58a3","Type":"ContainerStarted","Data":"b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0"}
Oct 30 22:07:15.709721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.709561    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:15.709721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.709583    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" event={"ID":"69963d5e-752a-4fe1-835e-23f0a83a58a3","Type":"ContainerStarted","Data":"1af720e4f52835769d61b77c6873ff24954072f52c07fd926e537e802094b310"}
Oct 30 22:07:15.762849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.762802    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused" start-of-body=
Oct 30 22:07:15.763065 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.762868    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused"
Oct 30 22:07:15.763643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.763570    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht" podStartSLOduration=2.763554218 podStartE2EDuration="2.763554218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:15.763357504 +0000 UTC m=+3418.685815802" watchObservedRunningTime="2026-10-30 22:07:15.763554218 +0000 UTC m=+3418.686012517"
Oct 30 22:07:15.765330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.765297    6695 patch_prober.go:29] interesting pod/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused" start-of-body=
Oct 30 22:07:15.765412 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.765358    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused"
Oct 30 22:07:15.778675 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.778641    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:07:15.802233 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:15.802059    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" podStartSLOduration=2.802042397 podStartE2EDuration="2.802042397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:15.800223266 +0000 UTC m=+3418.722681575" watchObservedRunningTime="2026-10-30 22:07:15.802042397 +0000 UTC m=+3418.724500695"
Oct 30 22:07:16.578184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.578093    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 22:07:16.605691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.605647    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume\") pod \"6857618f-f853-4559-83a3-8720a280c353\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") "
Oct 30 22:07:16.605861 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.605786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m\" (UniqueName: \"kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m\") pod \"6857618f-f853-4559-83a3-8720a280c353\" (UID: \"6857618f-f853-4559-83a3-8720a280c353\") "
Oct 30 22:07:16.607010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.606976    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume" pod "6857618f-f853-4559-83a3-8720a280c353" (UID: "6857618f-f853-4559-83a3-8720a280c353"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:16.609773 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.609731    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m" pod "6857618f-f853-4559-83a3-8720a280c353" (UID: "6857618f-f853-4559-83a3-8720a280c353"). InnerVolumeSpecName "kube-api-access-ktx8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:16.688870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.688818    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 22:07:16.708725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.708679    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6857618f-f853-4559-83a3-8720a280c353-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:16.708725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.708713    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktx8m\" (UniqueName: \"kubernetes.io/projected/6857618f-f853-4559-83a3-8720a280c353-kube-api-access-ktx8m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:16.750210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.749722    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused" start-of-body=
Oct 30 22:07:16.750210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.749789    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused"
Oct 30 22:07:16.774517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.773648    6695 patch_prober.go:29] interesting pod/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused" start-of-body=
Oct 30 22:07:16.774517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.773731    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused"
Oct 30 22:07:16.786356 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.786309    6695 generic.go:356] "Generic (PLEG): container finished" podID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerID="919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261" exitCode=0
Oct 30 22:07:16.786899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.786363    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" event={"ID":"97034497-4f1e-4a41-9030-b2f4ab146e9f","Type":"ContainerDied","Data":"919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261"}
Oct 30 22:07:16.786899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.786378    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"
Oct 30 22:07:16.786899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.786467    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" event={"ID":"97034497-4f1e-4a41-9030-b2f4ab146e9f","Type":"ContainerDied","Data":"4934a99a70e4caf93d408fbc1cc3d612347608945f976a2061a766463b8fa27f"}
Oct 30 22:07:16.786899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.786502    6695 scope.go:122] "RemoveContainer" containerID="919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261"
Oct 30 22:07:16.789005 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.788976    6695 generic.go:356] "Generic (PLEG): container finished" podID="6857618f-f853-4559-83a3-8720a280c353" containerID="dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551" exitCode=0
Oct 30 22:07:16.790072 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.790053    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"
Oct 30 22:07:16.790551 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.790517    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" event={"ID":"6857618f-f853-4559-83a3-8720a280c353","Type":"ContainerDied","Data":"dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551"}
Oct 30 22:07:16.790680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.790584    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" event={"ID":"6857618f-f853-4559-83a3-8720a280c353","Type":"ContainerDied","Data":"ed829e61f514e2821da1b1eee83ba20009af9a4952b85c1768d7edf39f76b494"}
Oct 30 22:07:16.802946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.802859    6695 scope.go:122] "RemoveContainer" containerID="919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261"
Oct 30 22:07:16.803349 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:07:16.803313    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261\": container with ID starting with 919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261 not found: ID does not exist" containerID="919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261"
Oct 30 22:07:16.803442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.803363    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261"} err="failed to get container status \"919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261\": rpc error: code = NotFound desc = could not find container \"919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261\": container with ID starting with 919d92de76f049bce9502dc9b38181bfd5b39da0374571c250342c645fd06261 not found: ID does not exist"
Oct 30 22:07:16.803442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.803389    6695 scope.go:122] "RemoveContainer" containerID="dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551"
Oct 30 22:07:16.810128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.810097    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl\" (UniqueName: \"kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl\") pod \"97034497-4f1e-4a41-9030-b2f4ab146e9f\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") "
Oct 30 22:07:16.810324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.810159    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume\") pod \"97034497-4f1e-4a41-9030-b2f4ab146e9f\" (UID: \"97034497-4f1e-4a41-9030-b2f4ab146e9f\") "
Oct 30 22:07:16.834731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.824911    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume" pod "97034497-4f1e-4a41-9030-b2f4ab146e9f" (UID: "97034497-4f1e-4a41-9030-b2f4ab146e9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:16.945628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.932114    6695 scope.go:122] "RemoveContainer" containerID="dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551"
Oct 30 22:07:16.945628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.939591    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97034497-4f1e-4a41-9030-b2f4ab146e9f-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:16.955659 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:07:16.946409    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551\": container with ID starting with dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551 not found: ID does not exist" containerID="dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551"
Oct 30 22:07:16.955659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:16.946459    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551"} err="failed to get container status \"dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551\": rpc error: code = NotFound desc = could not find container \"dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551\": container with ID starting with dbd1baaa10fcbcdfb7c34ac415e4f5f9d632b08b81df2c8d9d7ad63e65b2e551 not found: ID does not exist"
Oct 30 22:07:17.064960 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.064907    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl" pod "97034497-4f1e-4a41-9030-b2f4ab146e9f" (UID: "97034497-4f1e-4a41-9030-b2f4ab146e9f"). InnerVolumeSpecName "kube-api-access-jxhkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:17.081288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.081228    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"]
Oct 30 22:07:17.085296 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.085002    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d"]
Oct 30 22:07:17.164709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.164671    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhkl\" (UniqueName: \"kubernetes.io/projected/97034497-4f1e-4a41-9030-b2f4ab146e9f-kube-api-access-jxhkl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:17.169166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.169129    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"]
Oct 30 22:07:17.173576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.173533    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c"]
Oct 30 22:07:17.267525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.267484    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6857618f-f853-4559-83a3-8720a280c353" path="/var/lib/kubelet/pods/6857618f-f853-4559-83a3-8720a280c353/volumes"
Oct 30 22:07:17.268082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.268060    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" path="/var/lib/kubelet/pods/97034497-4f1e-4a41-9030-b2f4ab146e9f/volumes"
Oct 30 22:07:17.372904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.372809    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-49738b-5b94c8bd5-xx88d" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.55:50052\" within 1s: context deadline exceeded"
Oct 30 22:07:17.373158 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.372806    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-49738b-5f455d5955-7qs5c" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.54:50052\" within 1s: context deadline exceeded"
Oct 30 22:07:17.373258 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:17.372894    6695 logging.go:55] [core] [Channel #18222 SubChannel #18223]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.54:50052", ServerName: "10.134.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.54:50052: operation was canceled"
Oct 30 22:07:17.373333 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:17.372905    6695 logging.go:55] [core] [Channel #18220 SubChannel #18221]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.55:50052", ServerName: "10.134.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.55:50052: operation was canceled"
Oct 30 22:07:17.753696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.740997    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused" start-of-body=
Oct 30 22:07:17.753696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.741060    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused"
Oct 30 22:07:17.766699 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.765117    6695 patch_prober.go:29] interesting pod/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused" start-of-body=
Oct 30 22:07:17.766699 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.765188    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused"
Oct 30 22:07:17.962627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:17.962038    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:07:18.754631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:18.742092    6695 patch_prober.go:29] interesting pod/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused" start-of-body=
Oct 30 22:07:18.754631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:18.742162    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.190:15021/healthz/ready\": dial tcp 10.134.1.190:15021: connect: connection refused"
Oct 30 22:07:18.773626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:18.765866    6695 patch_prober.go:29] interesting pod/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused" start-of-body=
Oct 30 22:07:18.773626 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:18.765937    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.189:15021/healthz/ready\": dial tcp 10.134.1.189:15021: connect: connection refused"
Oct 30 22:07:19.745553 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:19.745515    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:07:19.746972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:19.746948    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:07:19.771640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:19.771569    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:07:19.772912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:19.772883    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:07:25.853168 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.853126    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"]
Oct 30 22:07:25.854745 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.854714    6695 cpu_manager.go:414] "containerMap: removing container" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing"
Oct 30 22:07:25.854745 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.854743    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing"
Oct 30 22:07:25.854926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.854816    6695 cpu_manager.go:414] "containerMap: removing container" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing"
Oct 30 22:07:25.854926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.854823    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing"
Oct 30 22:07:25.855007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.854957    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="97034497-4f1e-4a41-9030-b2f4ab146e9f" containerName="payload-pre-processing"
Oct 30 22:07:25.855007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.854973    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="6857618f-f853-4559-83a3-8720a280c353" containerName="payload-processing"
Oct 30 22:07:25.858851 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.858829    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:25.885002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.884966    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6537e6\"/\"openshift-service-ca.crt\""
Oct 30 22:07:25.885343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.885311    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6537e6\"/\"kube-root-ca.crt\""
Oct 30 22:07:25.885504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.884966    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6537e6\"/\"default-dockercfg-b76nq\""
Oct 30 22:07:25.885652 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.885501    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6537e6\"/\"test-model-6537e6-kserve-self-signed-certs\""
Oct 30 22:07:25.893216 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.893185    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"]
Oct 30 22:07:25.984044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.983999    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbf2w\" (UniqueName: \"kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:25.984264 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.984140    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:25.984264 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.984206    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:25.984451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.984268    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:25.984451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.984344    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:25.984451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:25.984391    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.085994    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbf2w\" (UniqueName: \"kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086070    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086097    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086182    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086356    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086466    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086574 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086548    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086720    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.086843 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.086824    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.118685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.112313    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.131627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.122917    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.165627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.165143    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbf2w\" (UniqueName: \"kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w\") pod \"test-model-6537e6-kserve-57f5bf4d67-n9stl\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") " pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.174858 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.173923    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:26.565352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.565319    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"]
Oct 30 22:07:26.566381 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:26.566355    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f0c618f_5bd8_4144_a480_f1c40234ce00.slice/crio-df5e31abb45f84eba84f701277c768d9d31349f29d62e81312db744fca612842 WatchSource:0}: Error finding container df5e31abb45f84eba84f701277c768d9d31349f29d62e81312db744fca612842: Status 404 returned error can't find the container with id df5e31abb45f84eba84f701277c768d9d31349f29d62e81312db744fca612842
Oct 30 22:07:26.793820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.793786    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:07:26.794020 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.793875    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:07:26.926760 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.926665    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl" event={"ID":"0f0c618f-5bd8-4144-a480-f1c40234ce00","Type":"ContainerStarted","Data":"ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164"}
Oct 30 22:07:26.926760 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.926719    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl" event={"ID":"0f0c618f-5bd8-4144-a480-f1c40234ce00","Type":"ContainerStarted","Data":"df5e31abb45f84eba84f701277c768d9d31349f29d62e81312db744fca612842"}
Oct 30 22:07:26.966461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:26.966400    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl" podStartSLOduration=1.966385828 podStartE2EDuration="1.966385828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:26.965730817 +0000 UTC m=+3429.888189126" watchObservedRunningTime="2026-10-30 22:07:26.966385828 +0000 UTC m=+3429.888844126"
Oct 30 22:07:27.056996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.056957    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"]
Oct 30 22:07:27.062042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.062015    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.109985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.109952    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:27.111565 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.111537    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:07:27.111830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.111753    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:27.111941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.111897    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:27.135717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.125864    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"]
Oct 30 22:07:27.139720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.139684    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.147064 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.147023    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.147533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.147507    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bkr\" (UniqueName: \"kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.147794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.147752    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.148419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.148379    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"]
Oct 30 22:07:27.155644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.153244    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"]
Oct 30 22:07:27.183052 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.182966    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:27.255206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.255146    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.255481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.255288    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.255481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.255398    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bkr\" (UniqueName: \"kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.255481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.255435    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.255763 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.255740    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.255909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.255893    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkznf\" (UniqueName: \"kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.257169 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.257135    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.259981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.259949    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.276694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.276664    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bkr\" (UniqueName: \"kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr\") pod \"maas-api-revoke-sub-5a6d34d5-m25r2\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.357971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.357922    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.358164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.358103    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.358213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.358162    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkznf\" (UniqueName: \"kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.358784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.358756    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.360435 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.360409    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.378620 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.378575    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkznf\" (UniqueName: \"kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf\") pod \"maas-api-revoke-sub-67824d27-fhg8p\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.412227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.412190    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:27.479327 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.479284    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:27.729030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.729003    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"]
Oct 30 22:07:27.732996 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:27.732962    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297ec5e3_ffe8_4638_b2e5_db493c895abf.slice/crio-ad523c9843c7b46b141e9b07b5e91f345352df7663e19d8493320ae8bff8c5ba WatchSource:0}: Error finding container ad523c9843c7b46b141e9b07b5e91f345352df7663e19d8493320ae8bff8c5ba: Status 404 returned error can't find the container with id ad523c9843c7b46b141e9b07b5e91f345352df7663e19d8493320ae8bff8c5ba
Oct 30 22:07:27.873422 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.873390    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"]
Oct 30 22:07:27.877015 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:27.876981    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0f7459_61a7_4679_b0a9_3b9059b96413.slice/crio-ede21c49cab53c3bbede16e6801d529c517119ba7f0cd86245023acac52290d2 WatchSource:0}: Error finding container ede21c49cab53c3bbede16e6801d529c517119ba7f0cd86245023acac52290d2: Status 404 returned error can't find the container with id ede21c49cab53c3bbede16e6801d529c517119ba7f0cd86245023acac52290d2
Oct 30 22:07:27.937130 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.937086    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p" event={"ID":"ab0f7459-61a7-4679-b0a9-3b9059b96413","Type":"ContainerStarted","Data":"ede21c49cab53c3bbede16e6801d529c517119ba7f0cd86245023acac52290d2"}
Oct 30 22:07:27.938409 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:27.938384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2" event={"ID":"297ec5e3-ffe8-4638-b2e5-db493c895abf","Type":"ContainerStarted","Data":"ad523c9843c7b46b141e9b07b5e91f345352df7663e19d8493320ae8bff8c5ba"}
Oct 30 22:07:28.949721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:28.949685    6695 generic.go:356] "Generic (PLEG): container finished" podID="ab0f7459-61a7-4679-b0a9-3b9059b96413" containerID="dc7dd0c2beef819152698c115604c8bdaa84ed5ce9b03f46010ac75000fd9354" exitCode=0
Oct 30 22:07:28.950149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:28.949769    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p" event={"ID":"ab0f7459-61a7-4679-b0a9-3b9059b96413","Type":"ContainerDied","Data":"dc7dd0c2beef819152698c115604c8bdaa84ed5ce9b03f46010ac75000fd9354"}
Oct 30 22:07:28.951516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:28.951489    6695 generic.go:356] "Generic (PLEG): container finished" podID="297ec5e3-ffe8-4638-b2e5-db493c895abf" containerID="84c760b8cc91598e017011ebdb9762a157898ff95f309e28cecd7c878f15b652" exitCode=0
Oct 30 22:07:28.951662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:28.951569    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2" event={"ID":"297ec5e3-ffe8-4638-b2e5-db493c895abf","Type":"ContainerDied","Data":"84c760b8cc91598e017011ebdb9762a157898ff95f309e28cecd7c878f15b652"}
Oct 30 22:07:28.984866 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:28.984828    6695 scope.go:122] "RemoveContainer" containerID="9790c610734f2db96d45d2d6354559c4c21646b9beab0ea7b7158bc634bf6b4e"
Oct 30 22:07:30.204664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.204628    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:30.206892 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.206874    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:30.361911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.361874    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca\") pod \"297ec5e3-ffe8-4638-b2e5-db493c895abf\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") "
Oct 30 22:07:30.362142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.361930    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr\" (UniqueName: \"kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr\") pod \"297ec5e3-ffe8-4638-b2e5-db493c895abf\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") "
Oct 30 22:07:30.362142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.362008    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf\" (UniqueName: \"kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf\") pod \"ab0f7459-61a7-4679-b0a9-3b9059b96413\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") "
Oct 30 22:07:30.362142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.362041    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca\") pod \"297ec5e3-ffe8-4638-b2e5-db493c895abf\" (UID: \"297ec5e3-ffe8-4638-b2e5-db493c895abf\") "
Oct 30 22:07:30.362142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.362130    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca\") pod \"ab0f7459-61a7-4679-b0a9-3b9059b96413\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") "
Oct 30 22:07:30.362379 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.362260    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca\") pod \"ab0f7459-61a7-4679-b0a9-3b9059b96413\" (UID: \"ab0f7459-61a7-4679-b0a9-3b9059b96413\") "
Oct 30 22:07:30.362590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.362558    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca" pod "297ec5e3-ffe8-4638-b2e5-db493c895abf" (UID: "297ec5e3-ffe8-4638-b2e5-db493c895abf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:30.362766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.362680    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca" pod "ab0f7459-61a7-4679-b0a9-3b9059b96413" (UID: "ab0f7459-61a7-4679-b0a9-3b9059b96413"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:30.363156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.363131    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:30.363242 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.363163    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:30.364282 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.364258    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca" pod "297ec5e3-ffe8-4638-b2e5-db493c895abf" (UID: "297ec5e3-ffe8-4638-b2e5-db493c895abf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:30.364373 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.364260    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca" pod "ab0f7459-61a7-4679-b0a9-3b9059b96413" (UID: "ab0f7459-61a7-4679-b0a9-3b9059b96413"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:30.364438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.364375    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf" pod "ab0f7459-61a7-4679-b0a9-3b9059b96413" (UID: "ab0f7459-61a7-4679-b0a9-3b9059b96413"). InnerVolumeSpecName "kube-api-access-hkznf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:30.364438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.364415    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr" pod "297ec5e3-ffe8-4638-b2e5-db493c895abf" (UID: "297ec5e3-ffe8-4638-b2e5-db493c895abf"). InnerVolumeSpecName "kube-api-access-57bkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:30.464702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.464665    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab0f7459-61a7-4679-b0a9-3b9059b96413-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:30.464702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.464698    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/297ec5e3-ffe8-4638-b2e5-db493c895abf-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:30.464702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.464707    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57bkr\" (UniqueName: \"kubernetes.io/projected/297ec5e3-ffe8-4638-b2e5-db493c895abf-kube-api-access-57bkr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:30.464981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.464718    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkznf\" (UniqueName: \"kubernetes.io/projected/ab0f7459-61a7-4679-b0a9-3b9059b96413-kube-api-access-hkznf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:30.987385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.987350    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"
Oct 30 22:07:30.987385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.987378    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p" event={"ID":"ab0f7459-61a7-4679-b0a9-3b9059b96413","Type":"ContainerDied","Data":"ede21c49cab53c3bbede16e6801d529c517119ba7f0cd86245023acac52290d2"}
Oct 30 22:07:30.987681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.987458    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ede21c49cab53c3bbede16e6801d529c517119ba7f0cd86245023acac52290d2"
Oct 30 22:07:30.989302 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.989273    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"
Oct 30 22:07:30.989478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.989454    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2" event={"ID":"297ec5e3-ffe8-4638-b2e5-db493c895abf","Type":"ContainerDied","Data":"ad523c9843c7b46b141e9b07b5e91f345352df7663e19d8493320ae8bff8c5ba"}
Oct 30 22:07:30.989658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:30.989592    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad523c9843c7b46b141e9b07b5e91f345352df7663e19d8493320ae8bff8c5ba"
Oct 30 22:07:31.077930 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:07:31.077879    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:07:31.094344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.094297    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"]
Oct 30 22:07:31.095317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.095291    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ab0f7459-61a7-4679-b0a9-3b9059b96413" containerName="revoke-keys"
Oct 30 22:07:31.095317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.095316    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0f7459-61a7-4679-b0a9-3b9059b96413" containerName="revoke-keys"
Oct 30 22:07:31.095517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.095339    6695 cpu_manager.go:414] "containerMap: removing container" podUID="297ec5e3-ffe8-4638-b2e5-db493c895abf" containerName="revoke-keys"
Oct 30 22:07:31.095517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.095347    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297ec5e3-ffe8-4638-b2e5-db493c895abf" containerName="revoke-keys"
Oct 30 22:07:31.095649 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.095581    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0f7459-61a7-4679-b0a9-3b9059b96413" containerName="revoke-keys"
Oct 30 22:07:31.095649 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.095620    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="297ec5e3-ffe8-4638-b2e5-db493c895abf" containerName="revoke-keys"
Oct 30 22:07:31.100241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.100209    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.136410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.127236    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a42c31\"/\"openshift-service-ca.crt\""
Oct 30 22:07:31.136410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.127888    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a42c31\"/\"test-model-a42c31-kserve-self-signed-certs\""
Oct 30 22:07:31.147964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.147913    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"]
Oct 30 22:07:31.178174 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.178132    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.178174 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.178174    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.178423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.178291    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.178595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.178510    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sz4\" (UniqueName: \"kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.178832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.178797    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.178988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.178860    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.222345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.222309    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a42c31\"/\"default-dockercfg-hs52s\""
Oct 30 22:07:31.226436 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.226409    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a42c31\"/\"kube-root-ca.crt\""
Oct 30 22:07:31.280570 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.280533    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.280770 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.280585    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.280770 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.280751    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.280770 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.280769    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.281208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.280997    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.281208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.281122    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sz4\" (UniqueName: \"kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.281208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.281143    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.281495 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.281205    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.281553 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.281077    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.283143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.283122    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.284157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.284102    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.329159 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.329116    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sz4\" (UniqueName: \"kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4\") pod \"test-model-a42c31-kserve-5dd47ccbc5-hphr8\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") " pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.417239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.417195    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:31.896665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:31.896629    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"]
Oct 30 22:07:31.899874 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:31.899839    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe77f2d_c2a6_4bff_9ee2_724ac5be6f1a.slice/crio-52c015654ca891810cacdd305dadb31f1a6a0f37d8626d1932c1cd6e376ee414 WatchSource:0}: Error finding container 52c015654ca891810cacdd305dadb31f1a6a0f37d8626d1932c1cd6e376ee414: Status 404 returned error can't find the container with id 52c015654ca891810cacdd305dadb31f1a6a0f37d8626d1932c1cd6e376ee414
Oct 30 22:07:32.041721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:32.041670    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8" event={"ID":"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a","Type":"ContainerStarted","Data":"52c015654ca891810cacdd305dadb31f1a6a0f37d8626d1932c1cd6e376ee414"}
Oct 30 22:07:33.052890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:33.052830    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8" event={"ID":"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a","Type":"ContainerStarted","Data":"99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3"}
Oct 30 22:07:33.090108 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:33.090048    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8" podStartSLOduration=2.090031825 podStartE2EDuration="2.090031825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:33.089817207 +0000 UTC m=+3436.012275507" watchObservedRunningTime="2026-10-30 22:07:33.090031825 +0000 UTC m=+3436.012490122"
Oct 30 22:07:33.417583 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:33.417480    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:34.346476 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.346439    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"]
Oct 30 22:07:34.351778 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.351752    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.377088 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.377056    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:34.379754 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.379726    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:07:34.379893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.379752    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:34.379893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.379764    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:34.382549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.382519    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"]
Oct 30 22:07:34.479635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.476260    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.479635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.476505    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzgmj\" (UniqueName: \"kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.479635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.476544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.578286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.578250    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzgmj\" (UniqueName: \"kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.578286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.578293    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.578566 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.578486    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.579141 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.579119    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.580867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.580836    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.597139 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.597069    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzgmj\" (UniqueName: \"kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj\") pod \"maas-api-revoke-sub-517f28b1-fwxzk\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.668469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.668430    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:34.984233 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:34.984203    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"]
Oct 30 22:07:34.987449 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:34.987415    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f38dca9_a36a_400b_b5a8_56c085719157.slice/crio-07806e27d87645ad1e47e472048eab041a51aa95682bde7f6f11989bcb9c8133 WatchSource:0}: Error finding container 07806e27d87645ad1e47e472048eab041a51aa95682bde7f6f11989bcb9c8133: Status 404 returned error can't find the container with id 07806e27d87645ad1e47e472048eab041a51aa95682bde7f6f11989bcb9c8133
Oct 30 22:07:35.079423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:35.079384    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk" event={"ID":"4f38dca9-a36a-400b-b5a8-56c085719157","Type":"ContainerStarted","Data":"07806e27d87645ad1e47e472048eab041a51aa95682bde7f6f11989bcb9c8133"}
Oct 30 22:07:36.093291 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:36.093254    6695 generic.go:356] "Generic (PLEG): container finished" podID="4f38dca9-a36a-400b-b5a8-56c085719157" containerID="74a6ed220be4e44354b7242a5f960c6e65357cedefaa3f335ee613c8fa0061f1" exitCode=0
Oct 30 22:07:36.093734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:36.093344    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk" event={"ID":"4f38dca9-a36a-400b-b5a8-56c085719157","Type":"ContainerDied","Data":"74a6ed220be4e44354b7242a5f960c6e65357cedefaa3f335ee613c8fa0061f1"}
Oct 30 22:07:36.181872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:36.181830    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:36.196873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:36.196779    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:37.114221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.114184    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:07:37.322540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.322513    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:37.451221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.451131    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca\") pod \"4f38dca9-a36a-400b-b5a8-56c085719157\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") "
Oct 30 22:07:37.451373 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.451297    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca\") pod \"4f38dca9-a36a-400b-b5a8-56c085719157\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") "
Oct 30 22:07:37.451427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.451388    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj\" (UniqueName: \"kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj\") pod \"4f38dca9-a36a-400b-b5a8-56c085719157\" (UID: \"4f38dca9-a36a-400b-b5a8-56c085719157\") "
Oct 30 22:07:37.451742 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.451715    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca" pod "4f38dca9-a36a-400b-b5a8-56c085719157" (UID: "4f38dca9-a36a-400b-b5a8-56c085719157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:37.452033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.452015    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:37.453328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.453298    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca" pod "4f38dca9-a36a-400b-b5a8-56c085719157" (UID: "4f38dca9-a36a-400b-b5a8-56c085719157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:37.453474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.453449    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj" pod "4f38dca9-a36a-400b-b5a8-56c085719157" (UID: "4f38dca9-a36a-400b-b5a8-56c085719157"). InnerVolumeSpecName "kube-api-access-vzgmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:37.553432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.553384    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzgmj\" (UniqueName: \"kubernetes.io/projected/4f38dca9-a36a-400b-b5a8-56c085719157-kube-api-access-vzgmj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:37.553432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:37.553424    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f38dca9-a36a-400b-b5a8-56c085719157-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:38.113332 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:38.113296    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"
Oct 30 22:07:38.113554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:38.113330    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk" event={"ID":"4f38dca9-a36a-400b-b5a8-56c085719157","Type":"ContainerDied","Data":"07806e27d87645ad1e47e472048eab041a51aa95682bde7f6f11989bcb9c8133"}
Oct 30 22:07:38.113554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:38.113391    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07806e27d87645ad1e47e472048eab041a51aa95682bde7f6f11989bcb9c8133"
Oct 30 22:07:41.418504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.418461    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:41.431290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.431259    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:41.846990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.846946    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"]
Oct 30 22:07:41.848301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.848264    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4f38dca9-a36a-400b-b5a8-56c085719157" containerName="revoke-keys"
Oct 30 22:07:41.848301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.848298    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f38dca9-a36a-400b-b5a8-56c085719157" containerName="revoke-keys"
Oct 30 22:07:41.848567 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.848550    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f38dca9-a36a-400b-b5a8-56c085719157" containerName="revoke-keys"
Oct 30 22:07:41.853943 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.853906    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:41.871867 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.871829    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19a725\"/\"openshift-service-ca.crt\""
Oct 30 22:07:41.878175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.878130    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"]
Oct 30 22:07:41.881388 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.881340    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19a725\"/\"test-model-19a725-kserve-self-signed-certs\""
Oct 30 22:07:41.881571 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.881437    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-19a725\"/\"kube-root-ca.crt\""
Oct 30 22:07:41.881571 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.881425    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-19a725\"/\"default-dockercfg-c6w6r\""
Oct 30 22:07:41.951072 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.951017    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:41.951281 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.951107    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:41.951350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.951326    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:41.951464 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.951448    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:41.951568 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.951546    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpzv\" (UniqueName: \"kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:41.951694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:41.951674    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.053621 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.053572    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpzv\" (UniqueName: \"kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.053862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.053677    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.053862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.053739    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.053862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.053782    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.054038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.053922    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.054085 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.054039    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.054207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.054177    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.054300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.054280    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.054410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.054390    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.056133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.056111    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.056568 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.056543    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.073717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.073680    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpzv\" (UniqueName: \"kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv\") pod \"test-model-19a725-kserve-7fbbcc5785-sgbfj\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") " pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.181686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.180818    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:42.275446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.275405    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:07:42.554985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:42.554944    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"]
Oct 30 22:07:42.557982 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:42.557949    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5edf70a6_ef3f_4ca3_93c5_d3219422a139.slice/crio-7463bea41fceaa7b408110ef233a4a354b66ea6bf2afca26b9cd9327b1275d04 WatchSource:0}: Error finding container 7463bea41fceaa7b408110ef233a4a354b66ea6bf2afca26b9cd9327b1275d04: Status 404 returned error can't find the container with id 7463bea41fceaa7b408110ef233a4a354b66ea6bf2afca26b9cd9327b1275d04
Oct 30 22:07:43.189692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:43.189637    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj" event={"ID":"5edf70a6-ef3f-4ca3-93c5-d3219422a139","Type":"ContainerStarted","Data":"ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8"}
Oct 30 22:07:43.189893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:43.189704    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj" event={"ID":"5edf70a6-ef3f-4ca3-93c5-d3219422a139","Type":"ContainerStarted","Data":"7463bea41fceaa7b408110ef233a4a354b66ea6bf2afca26b9cd9327b1275d04"}
Oct 30 22:07:43.230137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:43.230074    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj" podStartSLOduration=2.230054753 podStartE2EDuration="2.230054753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:43.226813988 +0000 UTC m=+3446.149272286" watchObservedRunningTime="2026-10-30 22:07:43.230054753 +0000 UTC m=+3446.152513051"
Oct 30 22:07:44.181643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:44.181565    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:45.270472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.270431    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"]
Oct 30 22:07:45.276101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.276071    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.293174 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.293137    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"]
Oct 30 22:07:45.299187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.299153    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:45.299347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.299198    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:45.303328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.303303    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:45.303461 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.303344    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:07:45.331584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.331533    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.331812 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.331765    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fcn\" (UniqueName: \"kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.332476 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.332441    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.434414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.434370    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.434622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.434438    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.434622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.434481    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fcn\" (UniqueName: \"kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.435208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.435182    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.436962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.436931    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.455076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.455045    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fcn\" (UniqueName: \"kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn\") pod \"maas-api-revoke-sub-5ae56c94-h28s9\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.600313 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.600221    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:45.918011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:45.917980    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"]
Oct 30 22:07:45.920820 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:07:45.920783    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34edc899_f332_4e5a_97ea_d0214e1abe69.slice/crio-c859f94f0987eb0b82ba9d51d1112b0584be4f194ed79a31e3cb8acc670a8c09 WatchSource:0}: Error finding container c859f94f0987eb0b82ba9d51d1112b0584be4f194ed79a31e3cb8acc670a8c09: Status 404 returned error can't find the container with id c859f94f0987eb0b82ba9d51d1112b0584be4f194ed79a31e3cb8acc670a8c09
Oct 30 22:07:46.219771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:46.219662    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9" event={"ID":"34edc899-f332-4e5a-97ea-d0214e1abe69","Type":"ContainerStarted","Data":"c859f94f0987eb0b82ba9d51d1112b0584be4f194ed79a31e3cb8acc670a8c09"}
Oct 30 22:07:47.229819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:47.229728    6695 generic.go:356] "Generic (PLEG): container finished" podID="34edc899-f332-4e5a-97ea-d0214e1abe69" containerID="38734b380bfb52f36f7ad7577fe43e026ff3d705d7e74cb2f5ec2ed8db177740" exitCode=0
Oct 30 22:07:47.229819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:47.229803    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9" event={"ID":"34edc899-f332-4e5a-97ea-d0214e1abe69","Type":"ContainerDied","Data":"38734b380bfb52f36f7ad7577fe43e026ff3d705d7e74cb2f5ec2ed8db177740"}
Oct 30 22:07:48.454671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.454586    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:48.593243 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.593196    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn\" (UniqueName: \"kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn\") pod \"34edc899-f332-4e5a-97ea-d0214e1abe69\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") "
Oct 30 22:07:48.593616 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.593580    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca\") pod \"34edc899-f332-4e5a-97ea-d0214e1abe69\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") "
Oct 30 22:07:48.594030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.594011    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca\") pod \"34edc899-f332-4e5a-97ea-d0214e1abe69\" (UID: \"34edc899-f332-4e5a-97ea-d0214e1abe69\") "
Oct 30 22:07:48.595918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.595887    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca" pod "34edc899-f332-4e5a-97ea-d0214e1abe69" (UID: "34edc899-f332-4e5a-97ea-d0214e1abe69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:48.615309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.615261    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn" pod "34edc899-f332-4e5a-97ea-d0214e1abe69" (UID: "34edc899-f332-4e5a-97ea-d0214e1abe69"). InnerVolumeSpecName "kube-api-access-84fcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:48.619567 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.619529    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca" pod "34edc899-f332-4e5a-97ea-d0214e1abe69" (UID: "34edc899-f332-4e5a-97ea-d0214e1abe69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:48.747537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.745832    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:48.747537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.745874    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84fcn\" (UniqueName: \"kubernetes.io/projected/34edc899-f332-4e5a-97ea-d0214e1abe69-kube-api-access-84fcn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:48.747537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:48.745889    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34edc899-f332-4e5a-97ea-d0214e1abe69-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:07:49.275379 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:49.275340    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"
Oct 30 22:07:49.355339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:49.355296    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9" event={"ID":"34edc899-f332-4e5a-97ea-d0214e1abe69","Type":"ContainerDied","Data":"c859f94f0987eb0b82ba9d51d1112b0584be4f194ed79a31e3cb8acc670a8c09"}
Oct 30 22:07:49.355666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:49.355640    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c859f94f0987eb0b82ba9d51d1112b0584be4f194ed79a31e3cb8acc670a8c09"
Oct 30 22:07:52.181567 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:52.181523    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:52.195942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:52.195904    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:52.319683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:52.319647    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:07:57.998571 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:07:57.998527    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"]
Oct 30 22:08:01.078058 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:08:01.078017    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:08:02.999521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:02.999463    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c" podUID="8ee057f9-9151-482f-b782-f080deb1056a" containerName="payload-processing" containerID="cri-o://3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62" gracePeriod=25
Oct 30 22:08:03.301140 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.301101    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:08:03.432246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.432196    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume\") pod \"8ee057f9-9151-482f-b782-f080deb1056a\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") "
Oct 30 22:08:03.432441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.432286    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials\") pod \"8ee057f9-9151-482f-b782-f080deb1056a\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") "
Oct 30 22:08:03.432441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.432334    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume\") pod \"8ee057f9-9151-482f-b782-f080deb1056a\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") "
Oct 30 22:08:03.432617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.432574    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh\") pod \"8ee057f9-9151-482f-b782-f080deb1056a\" (UID: \"8ee057f9-9151-482f-b782-f080deb1056a\") "
Oct 30 22:08:03.432785 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.432740    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume" pod "8ee057f9-9151-482f-b782-f080deb1056a" (UID: "8ee057f9-9151-482f-b782-f080deb1056a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:03.432886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.432859    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume" pod "8ee057f9-9151-482f-b782-f080deb1056a" (UID: "8ee057f9-9151-482f-b782-f080deb1056a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:03.433386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.433361    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:03.433386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.433383    6695 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ee057f9-9151-482f-b782-f080deb1056a-routing-overlay-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:03.434791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.434756    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials" pod "8ee057f9-9151-482f-b782-f080deb1056a" (UID: "8ee057f9-9151-482f-b782-f080deb1056a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:03.434889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.434810    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh" pod "8ee057f9-9151-482f-b782-f080deb1056a" (UID: "8ee057f9-9151-482f-b782-f080deb1056a"). InnerVolumeSpecName "kube-api-access-dqbdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:03.525458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.525422    6695 generic.go:356] "Generic (PLEG): container finished" podID="8ee057f9-9151-482f-b782-f080deb1056a" containerID="3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62" exitCode=0
Oct 30 22:08:03.525686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.525478    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"
Oct 30 22:08:03.525686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.525506    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c" event={"ID":"8ee057f9-9151-482f-b782-f080deb1056a","Type":"ContainerDied","Data":"3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62"}
Oct 30 22:08:03.525686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.525571    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c" event={"ID":"8ee057f9-9151-482f-b782-f080deb1056a","Type":"ContainerDied","Data":"0f747a618b5131a3ba96cdf4d83d7f1ec5a12317e4491453f4d3f1466ee63a0f"}
Oct 30 22:08:03.525686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.525597    6695 scope.go:122] "RemoveContainer" containerID="3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62"
Oct 30 22:08:03.535234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.535203    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqbdh\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-kube-api-access-dqbdh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:03.535346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.535239    6695 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ee057f9-9151-482f-b782-f080deb1056a-provider-credentials\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:03.538945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.538920    6695 scope.go:122] "RemoveContainer" containerID="3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62"
Oct 30 22:08:03.539347 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:08:03.539318    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62\": container with ID starting with 3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62 not found: ID does not exist" containerID="3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62"
Oct 30 22:08:03.539488 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.539351    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62"} err="failed to get container status \"3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62\": rpc error: code = NotFound desc = could not find container \"3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62\": container with ID starting with 3ed335876c53234334ca61e08178fe28f71db837496c1bb7eadc618e5b399b62 not found: ID does not exist"
Oct 30 22:08:03.580705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.580662    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"]
Oct 30 22:08:03.586870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:03.586831    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-6bdd6cs7c"]
Oct 30 22:08:04.103239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.103196    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"]
Oct 30 22:08:04.104347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.104319    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8ee057f9-9151-482f-b782-f080deb1056a" containerName="payload-processing"
Oct 30 22:08:04.104347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.104347    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee057f9-9151-482f-b782-f080deb1056a" containerName="payload-processing"
Oct 30 22:08:04.104563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.104439    6695 cpu_manager.go:414] "containerMap: removing container" podUID="34edc899-f332-4e5a-97ea-d0214e1abe69" containerName="revoke-keys"
Oct 30 22:08:04.104563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.104450    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34edc899-f332-4e5a-97ea-d0214e1abe69" containerName="revoke-keys"
Oct 30 22:08:04.104700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.104684    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="34edc899-f332-4e5a-97ea-d0214e1abe69" containerName="revoke-keys"
Oct 30 22:08:04.104769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.104716    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee057f9-9151-482f-b782-f080deb1056a" containerName="payload-processing"
Oct 30 22:08:04.110622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.110575    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.128055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.128020    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:04.128383 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.128122    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:08:04.133215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.133175    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"]
Oct 30 22:08:04.138135 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.138089    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:04.138135 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.138101    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:04.252167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.252129    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.252380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.252356    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8l7\" (UniqueName: \"kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.252502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.252482    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.354719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.354586    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.354875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.354766    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.354937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.354900    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8l7\" (UniqueName: \"kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.355377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.355353    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.357318 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.357296    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.376897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.376861    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8l7\" (UniqueName: \"kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7\") pod \"maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.428401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.428365    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:04.746823 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:04.746788    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"]
Oct 30 22:08:04.749779 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:08:04.749743    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f492c0_c973_4d46_b265_47e263158d63.slice/crio-2d094e0b529806d23ddaced62ba5f16c93e403187bf587bf8531ca075c6bd05b WatchSource:0}: Error finding container 2d094e0b529806d23ddaced62ba5f16c93e403187bf587bf8531ca075c6bd05b: Status 404 returned error can't find the container with id 2d094e0b529806d23ddaced62ba5f16c93e403187bf587bf8531ca075c6bd05b
Oct 30 22:08:05.217512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:05.217466    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee057f9-9151-482f-b782-f080deb1056a" path="/var/lib/kubelet/pods/8ee057f9-9151-482f-b782-f080deb1056a/volumes"
Oct 30 22:08:05.549957 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:05.549915    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs" event={"ID":"d9f492c0-c973-4d46-b265-47e263158d63","Type":"ContainerStarted","Data":"89abea0952c57da23d61c34bb8d0904f5d49799d71248e6289d850c95c5bc967"}
Oct 30 22:08:05.550164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:05.549972    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs" event={"ID":"d9f492c0-c973-4d46-b265-47e263158d63","Type":"ContainerStarted","Data":"2d094e0b529806d23ddaced62ba5f16c93e403187bf587bf8531ca075c6bd05b"}
Oct 30 22:08:05.589511 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:05.589450    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs" podStartSLOduration=1.589422866 podStartE2EDuration="1.589422866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:05.585750514 +0000 UTC m=+3468.508208814" watchObservedRunningTime="2026-10-30 22:08:05.589422866 +0000 UTC m=+3468.511881164"
Oct 30 22:08:06.561543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:06.561505    6695 generic.go:356] "Generic (PLEG): container finished" podID="d9f492c0-c973-4d46-b265-47e263158d63" containerID="89abea0952c57da23d61c34bb8d0904f5d49799d71248e6289d850c95c5bc967" exitCode=0
Oct 30 22:08:06.562011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:06.561591    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs" event={"ID":"d9f492c0-c973-4d46-b265-47e263158d63","Type":"ContainerDied","Data":"89abea0952c57da23d61c34bb8d0904f5d49799d71248e6289d850c95c5bc967"}
Oct 30 22:08:08.040865 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.040818    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:08.175166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.175126    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7\" (UniqueName: \"kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7\") pod \"d9f492c0-c973-4d46-b265-47e263158d63\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") "
Oct 30 22:08:08.175479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.175455    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca\") pod \"d9f492c0-c973-4d46-b265-47e263158d63\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") "
Oct 30 22:08:08.175769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.175745    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca\") pod \"d9f492c0-c973-4d46-b265-47e263158d63\" (UID: \"d9f492c0-c973-4d46-b265-47e263158d63\") "
Oct 30 22:08:08.178000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.177971    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca" pod "d9f492c0-c973-4d46-b265-47e263158d63" (UID: "d9f492c0-c973-4d46-b265-47e263158d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:08.191658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.183562    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7" pod "d9f492c0-c973-4d46-b265-47e263158d63" (UID: "d9f492c0-c973-4d46-b265-47e263158d63"). InnerVolumeSpecName "kube-api-access-jn8l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:08.195227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.195186    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca" pod "d9f492c0-c973-4d46-b265-47e263158d63" (UID: "d9f492c0-c973-4d46-b265-47e263158d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:08.278483 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.278434    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn8l7\" (UniqueName: \"kubernetes.io/projected/d9f492c0-c973-4d46-b265-47e263158d63-kube-api-access-jn8l7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:08.278736 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.278500    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:08.278736 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.278538    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f492c0-c973-4d46-b265-47e263158d63-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:08.674123 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.674060    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs" event={"ID":"d9f492c0-c973-4d46-b265-47e263158d63","Type":"ContainerDied","Data":"2d094e0b529806d23ddaced62ba5f16c93e403187bf587bf8531ca075c6bd05b"}
Oct 30 22:08:08.674318 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.674184    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d094e0b529806d23ddaced62ba5f16c93e403187bf587bf8531ca075c6bd05b"
Oct 30 22:08:08.674318 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:08.674276    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"
Oct 30 22:08:11.052708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:11.052657    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"]
Oct 30 22:08:11.088047 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:11.088010    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9507eef9-95699"]
Oct 30 22:08:11.242646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:11.242591    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf589d7-66a3-495d-8e89-05ac79973334" path="/var/lib/kubelet/pods/fdf589d7-66a3-495d-8e89-05ac79973334/volumes"
Oct 30 22:08:14.619980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:14.619945    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"]
Oct 30 22:08:14.654513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:14.654471    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"]
Oct 30 22:08:16.762587 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:16.762548    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"]
Oct 30 22:08:16.774570 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:16.774533    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-49738b-lwmhd"]
Oct 30 22:08:17.220497 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:17.220458    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33719f3a-c049-49c5-bc1e-4d0114d6c1e1" path="/var/lib/kubelet/pods/33719f3a-c049-49c5-bc1e-4d0114d6c1e1/volumes"
Oct 30 22:08:18.037039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:18.036992    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"]
Oct 30 22:08:18.047562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:18.047524    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253a5db0-rrvh6"]
Oct 30 22:08:19.071770 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.071662    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"]
Oct 30 22:08:19.135629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.121587    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3d5d13-ncxhs"]
Oct 30 22:08:19.232679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.226418    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a2bdd5-7363-4526-b7a3-29f5b26c41d9" path="/var/lib/kubelet/pods/d9a2bdd5-7363-4526-b7a3-29f5b26c41d9/volumes"
Oct 30 22:08:19.250346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.248809    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f492c0-c973-4d46-b265-47e263158d63" path="/var/lib/kubelet/pods/d9f492c0-c973-4d46-b265-47e263158d63/volumes"
Oct 30 22:08:19.621078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.621032    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerName="payload-processing" containerID="cri-o://edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8" gracePeriod=25
Oct 30 22:08:19.758680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.758581    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerName="payload-pre-processing" containerID="cri-o://8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341" gracePeriod=25
Oct 30 22:08:19.836665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.836586    6695 generic.go:356] "Generic (PLEG): container finished" podID="fe820873-6dc6-44a7-8813-6902c183c856" containerID="304cf1235d6d2691f6e379c9f4b81120e0fcb7c6aa78b0487bcea5f42a910712" exitCode=0
Oct 30 22:08:19.836665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.836660    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" event={"ID":"fe820873-6dc6-44a7-8813-6902c183c856","Type":"ContainerDied","Data":"304cf1235d6d2691f6e379c9f4b81120e0fcb7c6aa78b0487bcea5f42a910712"}
Oct 30 22:08:19.837210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.837189    6695 scope.go:122] "RemoveContainer" containerID="304cf1235d6d2691f6e379c9f4b81120e0fcb7c6aa78b0487bcea5f42a910712"
Oct 30 22:08:19.839339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.839304    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:19.839450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.839391    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:19.839450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:19.839427    6695 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:08:20.854278 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:20.854228    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" event={"ID":"fe820873-6dc6-44a7-8813-6902c183c856","Type":"ContainerStarted","Data":"75df84fe3c12157a9963e417637e44c2aac700b5dbe67147f8f7dca61f6d83b9"}
Oct 30 22:08:21.631061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:21.631018    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:08:21.635909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:21.635867    6695 patch_prober.go:29] interesting pod/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused" start-of-body=
Oct 30 22:08:21.636147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:21.635933    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused"
Oct 30 22:08:22.055643 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:22.053434    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"]
Oct 30 22:08:22.071642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:22.068065    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3053b38e-v5dfw"]
Oct 30 22:08:22.612658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:22.612411    6695 patch_prober.go:29] interesting pod/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused" start-of-body=
Oct 30 22:08:22.612658 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:22.612474    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused"
Oct 30 22:08:23.255953 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.255911    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093f5235-281a-4726-8444-17047ac4d948" path="/var/lib/kubelet/pods/093f5235-281a-4726-8444-17047ac4d948/volumes"
Oct 30 22:08:23.615633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.613042    6695 patch_prober.go:29] interesting pod/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused" start-of-body=
Oct 30 22:08:23.615633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.613107    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused"
Oct 30 22:08:23.864286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.864244    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"]
Oct 30 22:08:23.865569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.865520    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy" containerID="cri-o://c2cfe4c2625709ca3053e5769f05bb7f750e147ba834e6d88bdf6032ef16d8dc" gracePeriod=30
Oct 30 22:08:23.881787 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.881742    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:23.882181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:23.882103    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:24.617630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:24.616754    6695 patch_prober.go:29] interesting pod/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused" start-of-body=
Oct 30 22:08:24.617630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:24.616868    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused"
Oct 30 22:08:25.574796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:25.574756    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:08:25.633631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:25.610698    6695 patch_prober.go:29] interesting pod/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused" start-of-body=
Oct 30 22:08:25.634369 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:25.634319    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused"
Oct 30 22:08:26.616640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:26.615530    6695 patch_prober.go:29] interesting pod/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused" start-of-body=
Oct 30 22:08:26.616640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:26.615622    6695 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.200:15021/healthz/ready\": dial tcp 10.134.1.200:15021: connect: connection refused"
Oct 30 22:08:27.014876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.014784    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"]
Oct 30 22:08:27.015761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.015743    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d9f492c0-c973-4d46-b265-47e263158d63" containerName="revoke-keys"
Oct 30 22:08:27.015823 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.015762    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f492c0-c973-4d46-b265-47e263158d63" containerName="revoke-keys"
Oct 30 22:08:27.015909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.015899    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f492c0-c973-4d46-b265-47e263158d63" containerName="revoke-keys"
Oct 30 22:08:27.020112 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.020087    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.038309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.038270    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"]
Oct 30 22:08:27.043998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.043966    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:08:27.044175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.044049    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:08:27.045790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.045760    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:27.045933 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.045823    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:08:27.117659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.117340    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjrm\" (UniqueName: \"kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.117659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.117515    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.117659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.117623    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.219344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.219304    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjrm\" (UniqueName: \"kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.219545 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.219407    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.219545 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.219451    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.220132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.220104    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.222040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.222020    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.239650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.239616    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjrm\" (UniqueName: \"kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm\") pod \"maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.337114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.337063    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:27.617912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.617829    6695 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:08:27.619275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.619248    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:08:27.651834 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:27.651793    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"]
Oct 30 22:08:27.654028 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:08:27.653976    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ede435_e4fe_4b54_960b_b60e113bde65.slice/crio-199182adc693ab22bfb571a5d84742f3c96688bdca1449bba0fd1ec0f816df2f WatchSource:0}: Error finding container 199182adc693ab22bfb571a5d84742f3c96688bdca1449bba0fd1ec0f816df2f: Status 404 returned error can't find the container with id 199182adc693ab22bfb571a5d84742f3c96688bdca1449bba0fd1ec0f816df2f
Oct 30 22:08:28.075538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:28.075498    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f" event={"ID":"83ede435-e4fe-4b54-960b-b60e113bde65","Type":"ContainerStarted","Data":"199182adc693ab22bfb571a5d84742f3c96688bdca1449bba0fd1ec0f816df2f"}
Oct 30 22:08:29.088915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.088880    6695 generic.go:356] "Generic (PLEG): container finished" podID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerID="c2cfe4c2625709ca3053e5769f05bb7f750e147ba834e6d88bdf6032ef16d8dc" exitCode=0
Oct 30 22:08:29.089346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.088967    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" event={"ID":"97f58aad-5421-4be1-ab2b-d26d0ceb46e5","Type":"ContainerDied","Data":"c2cfe4c2625709ca3053e5769f05bb7f750e147ba834e6d88bdf6032ef16d8dc"}
Oct 30 22:08:29.090989 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.090956    6695 generic.go:356] "Generic (PLEG): container finished" podID="83ede435-e4fe-4b54-960b-b60e113bde65" containerID="06cdc1c7aefc644af58a7d2bc7707640f6b980a172cf5be96f4fa7d3b1da9d32" exitCode=0
Oct 30 22:08:29.091137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.091034    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f" event={"ID":"83ede435-e4fe-4b54-960b-b60e113bde65","Type":"ContainerDied","Data":"06cdc1c7aefc644af58a7d2bc7707640f6b980a172cf5be96f4fa7d3b1da9d32"}
Oct 30 22:08:29.253153 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.253116    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:08:29.361687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361545    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.361687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361677    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.361949 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361712    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.361949 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361795    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.361949 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361848    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.361949 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361921    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.362198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.361999    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.362267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362164    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.362323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362273    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.362365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362319    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:29.362365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362325    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.362473 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362431    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket\") pod \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\" (UID: \"97f58aad-5421-4be1-ab2b-d26d0ceb46e5\") "
Oct 30 22:08:29.362706 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362598    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.362798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.362729    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.363365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.363340    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.363474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.363370    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.363474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.363386    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.363474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.363400    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.363474 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.363414    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.364790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.364527    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:29.364790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.364631    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:29.364790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.364735    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "kube-api-access-49v4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:29.364972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.364801    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy" pod "97f58aad-5421-4be1-ab2b-d26d0ceb46e5" (UID: "97f58aad-5421-4be1-ab2b-d26d0ceb46e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.464939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.464895    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.464939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.464934    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.464939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.464945    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49v4q\" (UniqueName: \"kubernetes.io/projected/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-kube-api-access-49v4q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.465210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:29.464955    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f58aad-5421-4be1-ab2b-d26d0ceb46e5-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:30.102513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.102475    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"
Oct 30 22:08:30.103132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.102484    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54" event={"ID":"97f58aad-5421-4be1-ab2b-d26d0ceb46e5","Type":"ContainerDied","Data":"032b5bf6b5c5b7333be0f510a99487eda7b6b7c45cdbce2f4a27c7cdfa2bead0"}
Oct 30 22:08:30.103132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.102664    6695 scope.go:122] "RemoveContainer" containerID="c2cfe4c2625709ca3053e5769f05bb7f750e147ba834e6d88bdf6032ef16d8dc"
Oct 30 22:08:30.170754 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.170503    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"]
Oct 30 22:08:30.177583 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.177545    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-58v54"]
Oct 30 22:08:30.291346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.291315    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:30.380802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.380689    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm\" (UniqueName: \"kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm\") pod \"83ede435-e4fe-4b54-960b-b60e113bde65\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") "
Oct 30 22:08:30.381017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.380937    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca\") pod \"83ede435-e4fe-4b54-960b-b60e113bde65\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") "
Oct 30 22:08:30.381166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.381145    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca\") pod \"83ede435-e4fe-4b54-960b-b60e113bde65\" (UID: \"83ede435-e4fe-4b54-960b-b60e113bde65\") "
Oct 30 22:08:30.381418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.381391    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca" pod "83ede435-e4fe-4b54-960b-b60e113bde65" (UID: "83ede435-e4fe-4b54-960b-b60e113bde65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:30.381883 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.381861    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:30.383429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.383401    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca" pod "83ede435-e4fe-4b54-960b-b60e113bde65" (UID: "83ede435-e4fe-4b54-960b-b60e113bde65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:30.383708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.383687    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm" pod "83ede435-e4fe-4b54-960b-b60e113bde65" (UID: "83ede435-e4fe-4b54-960b-b60e113bde65"). InnerVolumeSpecName "kube-api-access-bdjrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:30.484122 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.484072    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ede435-e4fe-4b54-960b-b60e113bde65-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:30.484122 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:30.484114    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjrm\" (UniqueName: \"kubernetes.io/projected/83ede435-e4fe-4b54-960b-b60e113bde65-kube-api-access-bdjrm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:31.076967 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:08:31.076921    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:08:31.117288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:31.117256    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"
Oct 30 22:08:31.117288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:31.117274    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f" event={"ID":"83ede435-e4fe-4b54-960b-b60e113bde65","Type":"ContainerDied","Data":"199182adc693ab22bfb571a5d84742f3c96688bdca1449bba0fd1ec0f816df2f"}
Oct 30 22:08:31.117745 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:31.117324    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199182adc693ab22bfb571a5d84742f3c96688bdca1449bba0fd1ec0f816df2f"
Oct 30 22:08:31.217651 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:31.217618    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" path="/var/lib/kubelet/pods/97f58aad-5421-4be1-ab2b-d26d0ceb46e5/volumes"
Oct 30 22:08:33.881750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:33.881706    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:33.882176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:33.881812    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:37.236234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:37.236194    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"]
Oct 30 22:08:37.275031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:37.274989    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"]
Oct 30 22:08:39.944798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:39.944763    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 22:08:40.012766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.012721    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume\") pod \"02f934b6-ab7e-4697-8041-e3923bbc98bc\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") "
Oct 30 22:08:40.013091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.013068    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp\" (UniqueName: \"kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp\") pod \"02f934b6-ab7e-4697-8041-e3923bbc98bc\" (UID: \"02f934b6-ab7e-4697-8041-e3923bbc98bc\") "
Oct 30 22:08:40.014067 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.014038    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume" pod "02f934b6-ab7e-4697-8041-e3923bbc98bc" (UID: "02f934b6-ab7e-4697-8041-e3923bbc98bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:40.014353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.014332    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f934b6-ab7e-4697-8041-e3923bbc98bc-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.017149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.017118    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp" pod "02f934b6-ab7e-4697-8041-e3923bbc98bc" (UID: "02f934b6-ab7e-4697-8041-e3923bbc98bc"). InnerVolumeSpecName "kube-api-access-r7glp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:40.052580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.052549    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 22:08:40.115043 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.115007    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h\" (UniqueName: \"kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h\") pod \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") "
Oct 30 22:08:40.115219 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.115184    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume\") pod \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\" (UID: \"26cf8de1-c1a9-4167-afe0-d50d84f0ca15\") "
Oct 30 22:08:40.115749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.115714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume" pod "26cf8de1-c1a9-4167-afe0-d50d84f0ca15" (UID: "26cf8de1-c1a9-4167-afe0-d50d84f0ca15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:40.116081 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.116056    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.116199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.116084    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7glp\" (UniqueName: \"kubernetes.io/projected/02f934b6-ab7e-4697-8041-e3923bbc98bc-kube-api-access-r7glp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.118283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.118249    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h" pod "26cf8de1-c1a9-4167-afe0-d50d84f0ca15" (UID: "26cf8de1-c1a9-4167-afe0-d50d84f0ca15"). InnerVolumeSpecName "kube-api-access-tm58h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:40.205692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.205586    6695 generic.go:356] "Generic (PLEG): container finished" podID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerID="8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341" exitCode=0
Oct 30 22:08:40.205692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.205635    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" event={"ID":"26cf8de1-c1a9-4167-afe0-d50d84f0ca15","Type":"ContainerDied","Data":"8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341"}
Oct 30 22:08:40.205692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.205681    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"
Oct 30 22:08:40.206039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.205706    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9" event={"ID":"26cf8de1-c1a9-4167-afe0-d50d84f0ca15","Type":"ContainerDied","Data":"395e764a8c0569656842a8d1be36fd8413b83994cd510476a2b687a478d1bc8c"}
Oct 30 22:08:40.206039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.205732    6695 scope.go:122] "RemoveContainer" containerID="8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341"
Oct 30 22:08:40.207412 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.207383    6695 generic.go:356] "Generic (PLEG): container finished" podID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerID="edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8" exitCode=0
Oct 30 22:08:40.207524 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.207432    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"
Oct 30 22:08:40.207524 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.207435    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" event={"ID":"02f934b6-ab7e-4697-8041-e3923bbc98bc","Type":"ContainerDied","Data":"edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8"}
Oct 30 22:08:40.207524 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.207492    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6" event={"ID":"02f934b6-ab7e-4697-8041-e3923bbc98bc","Type":"ContainerDied","Data":"9bc30b66c84d2448d3c6fdb0a91a8d8fb7d0408ec396a25639e0ac11637571d1"}
Oct 30 22:08:40.217455 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.217423    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm58h\" (UniqueName: \"kubernetes.io/projected/26cf8de1-c1a9-4167-afe0-d50d84f0ca15-kube-api-access-tm58h\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.219027 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.219002    6695 scope.go:122] "RemoveContainer" containerID="8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341"
Oct 30 22:08:40.219473 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:08:40.219439    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341\": container with ID starting with 8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341 not found: ID does not exist" containerID="8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341"
Oct 30 22:08:40.219576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.219484    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341"} err="failed to get container status \"8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341\": rpc error: code = NotFound desc = could not find container \"8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341\": container with ID starting with 8cc34e9c3857d5691b4ab75996ac800fc4e28b9d0d822e98af96588c3ddcc341 not found: ID does not exist"
Oct 30 22:08:40.219576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.219518    6695 scope.go:122] "RemoveContainer" containerID="edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8"
Oct 30 22:08:40.231862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.231837    6695 scope.go:122] "RemoveContainer" containerID="edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8"
Oct 30 22:08:40.232229 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:08:40.232205    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8\": container with ID starting with edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8 not found: ID does not exist" containerID="edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8"
Oct 30 22:08:40.232303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.232240    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8"} err="failed to get container status \"edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8\": rpc error: code = NotFound desc = could not find container \"edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8\": container with ID starting with edd4958cd41342f3f956833eeb794e01e8546efc9219f641a71ed685d89979a8 not found: ID does not exist"
Oct 30 22:08:40.271586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.271548    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"]
Oct 30 22:08:40.279494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.279444    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3d5d13-658f6d99ff-4smg9"]
Oct 30 22:08:40.310516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.310477    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"]
Oct 30 22:08:40.315386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:40.315343    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3d5d13-5bc6dd6d95-j6xj6"]
Oct 30 22:08:41.347683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:41.347353    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" path="/var/lib/kubelet/pods/02f934b6-ab7e-4697-8041-e3923bbc98bc/volumes"
Oct 30 22:08:41.375629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:41.364680    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" path="/var/lib/kubelet/pods/26cf8de1-c1a9-4167-afe0-d50d84f0ca15/volumes"
Oct 30 22:08:42.237359 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:42.237310    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" containerName="payload-processing" containerID="cri-o://e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14" gracePeriod=25
Oct 30 22:08:42.275353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:42.275307    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerName="payload-pre-processing" containerID="cri-o://36cbf9275f38ee9397962b1508da04926ddbd0940760e49cc329fa1c154a0128" gracePeriod=25
Oct 30 22:08:44.031912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:44.031856    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"]
Oct 30 22:08:44.042893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:44.042849    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a12d005-v4zvh"]
Oct 30 22:08:44.093366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:44.093317    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:45.100932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:45.100890    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:45.216911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:45.216873    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37995d5-3c94-4516-ae8c-6b57951041ed" path="/var/lib/kubelet/pods/a37995d5-3c94-4516-ae8c-6b57951041ed/volumes"
Oct 30 22:08:46.030543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:46.030494    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"]
Oct 30 22:08:46.041466 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:46.041430    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ed5937-6tkpn"]
Oct 30 22:08:47.218021 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:47.217987    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ac4b1b-8751-402e-8b17-b2fda048e437" path="/var/lib/kubelet/pods/71ac4b1b-8751-402e-8b17-b2fda048e437/volumes"
Oct 30 22:08:54.094055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:54.094017    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:08:55.106162 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:08:55.101913    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:01.080936 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:09:01.080815    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:09:02.650250 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.650205    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 22:09:02.683234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.683197    6695 generic.go:356] "Generic (PLEG): container finished" podID="c1702520-0360-4cbe-81dd-038186e3cb40" containerID="e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14" exitCode=0
Oct 30 22:09:02.683469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.683250    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"
Oct 30 22:09:02.683469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.683284    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" event={"ID":"c1702520-0360-4cbe-81dd-038186e3cb40","Type":"ContainerDied","Data":"e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14"}
Oct 30 22:09:02.683469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.683353    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps" event={"ID":"c1702520-0360-4cbe-81dd-038186e3cb40","Type":"ContainerDied","Data":"ff6334bf2e20dae9f0d4686eefb1a309f669d4d3d46578f32eac21cc2a5596fd"}
Oct 30 22:09:02.683469 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.683386    6695 scope.go:122] "RemoveContainer" containerID="e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14"
Oct 30 22:09:02.685101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.685072    6695 generic.go:356] "Generic (PLEG): container finished" podID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerID="36cbf9275f38ee9397962b1508da04926ddbd0940760e49cc329fa1c154a0128" exitCode=0
Oct 30 22:09:02.685226 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.685132    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" event={"ID":"0002ef74-d862-4bb3-9615-7ec896d884c4","Type":"ContainerDied","Data":"36cbf9275f38ee9397962b1508da04926ddbd0940760e49cc329fa1c154a0128"}
Oct 30 22:09:02.705173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.705144    6695 scope.go:122] "RemoveContainer" containerID="e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14"
Oct 30 22:09:02.705527 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:09:02.705501    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14\": container with ID starting with e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14 not found: ID does not exist" containerID="e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14"
Oct 30 22:09:02.705701 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.705534    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14"} err="failed to get container status \"e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14\": rpc error: code = NotFound desc = could not find container \"e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14\": container with ID starting with e462cfebd5f02431103c27530c8bbcb1fb2f81795541846e5f3e3581e1b18b14 not found: ID does not exist"
Oct 30 22:09:02.707547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.707522    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 22:09:02.781341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.781289    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume\") pod \"c1702520-0360-4cbe-81dd-038186e3cb40\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") "
Oct 30 22:09:02.781514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.781459    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq\" (UniqueName: \"kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq\") pod \"c1702520-0360-4cbe-81dd-038186e3cb40\" (UID: \"c1702520-0360-4cbe-81dd-038186e3cb40\") "
Oct 30 22:09:02.781856 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.781819    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume" pod "c1702520-0360-4cbe-81dd-038186e3cb40" (UID: "c1702520-0360-4cbe-81dd-038186e3cb40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:02.782438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.782406    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1702520-0360-4cbe-81dd-038186e3cb40-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:02.783862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.783833    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq" pod "c1702520-0360-4cbe-81dd-038186e3cb40" (UID: "c1702520-0360-4cbe-81dd-038186e3cb40"). InnerVolumeSpecName "kube-api-access-twzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:02.883866 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.883825    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844\" (UniqueName: \"kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844\") pod \"0002ef74-d862-4bb3-9615-7ec896d884c4\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") "
Oct 30 22:09:02.884071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.883892    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume\") pod \"0002ef74-d862-4bb3-9615-7ec896d884c4\" (UID: \"0002ef74-d862-4bb3-9615-7ec896d884c4\") "
Oct 30 22:09:02.884293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.884263    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume" pod "0002ef74-d862-4bb3-9615-7ec896d884c4" (UID: "0002ef74-d862-4bb3-9615-7ec896d884c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:02.884660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.884641    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0002ef74-d862-4bb3-9615-7ec896d884c4-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:02.884742 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.884665    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twzjq\" (UniqueName: \"kubernetes.io/projected/c1702520-0360-4cbe-81dd-038186e3cb40-kube-api-access-twzjq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:02.886142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.886115    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844" pod "0002ef74-d862-4bb3-9615-7ec896d884c4" (UID: "0002ef74-d862-4bb3-9615-7ec896d884c4"). InnerVolumeSpecName "kube-api-access-8b844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:02.987003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:02.986954    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b844\" (UniqueName: \"kubernetes.io/projected/0002ef74-d862-4bb3-9615-7ec896d884c4-kube-api-access-8b844\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:03.030016 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.029975    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"]
Oct 30 22:09:03.041661 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.041541    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdb168-5c75f6f6c5-7fhps"]
Oct 30 22:09:03.217586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.217548    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" path="/var/lib/kubelet/pods/c1702520-0360-4cbe-81dd-038186e3cb40/volumes"
Oct 30 22:09:03.698432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.698399    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"
Oct 30 22:09:03.698925 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.698430    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6" event={"ID":"0002ef74-d862-4bb3-9615-7ec896d884c4","Type":"ContainerDied","Data":"44d4c628fe2fd18c5ae8cebc1af13a1936443958fef3681a337384c42719b453"}
Oct 30 22:09:03.698925 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.698498    6695 scope.go:122] "RemoveContainer" containerID="36cbf9275f38ee9397962b1508da04926ddbd0940760e49cc329fa1c154a0128"
Oct 30 22:09:03.763137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.763068    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"]
Oct 30 22:09:03.774256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:03.774212    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdb168-777554f75f-gslv6"]
Oct 30 22:09:05.217456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:05.217368    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" path="/var/lib/kubelet/pods/0002ef74-d862-4bb3-9615-7ec896d884c4/volumes"
Oct 30 22:09:14.542700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:14.542660    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"]
Oct 30 22:09:14.551718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:14.551688    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fdb168-kvg2f"]
Oct 30 22:09:15.217850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:15.217809    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ede435-e4fe-4b54-960b-b60e113bde65" path="/var/lib/kubelet/pods/83ede435-e4fe-4b54-960b-b60e113bde65/volumes"
Oct 30 22:09:22.301493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.301449    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"]
Oct 30 22:09:22.302577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302550    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerName="payload-pre-processing"
Oct 30 22:09:22.302577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302577    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerName="payload-pre-processing"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302625    6695 cpu_manager.go:414] "containerMap: removing container" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerName="payload-processing"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302634    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerName="payload-processing"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302658    6695 cpu_manager.go:414] "containerMap: removing container" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302667    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302690    6695 cpu_manager.go:414] "containerMap: removing container" podUID="83ede435-e4fe-4b54-960b-b60e113bde65" containerName="revoke-keys"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302697    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ede435-e4fe-4b54-960b-b60e113bde65" containerName="revoke-keys"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302732    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" containerName="payload-processing"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302740    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" containerName="payload-processing"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302758    6695 cpu_manager.go:414] "containerMap: removing container" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerName="payload-pre-processing"
Oct 30 22:09:22.302819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302767    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerName="payload-pre-processing"
Oct 30 22:09:22.303347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.302986    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f58aad-5421-4be1-ab2b-d26d0ceb46e5" containerName="istio-proxy"
Oct 30 22:09:22.303347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.303005    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ede435-e4fe-4b54-960b-b60e113bde65" containerName="revoke-keys"
Oct 30 22:09:22.303347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.303023    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1702520-0360-4cbe-81dd-038186e3cb40" containerName="payload-processing"
Oct 30 22:09:22.303347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.303035    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cf8de1-c1a9-4167-afe0-d50d84f0ca15" containerName="payload-pre-processing"
Oct 30 22:09:22.303347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.303052    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0002ef74-d862-4bb3-9615-7ec896d884c4" containerName="payload-pre-processing"
Oct 30 22:09:22.303347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.303064    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f934b6-ab7e-4697-8041-e3923bbc98bc" containerName="payload-processing"
Oct 30 22:09:22.307293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.307268    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.321801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.321768    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:22.322186 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.322152    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:09:22.325790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.325758    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"]
Oct 30 22:09:22.331749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.331721    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:22.331749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.331750    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:22.385427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.385389    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmtc\" (UniqueName: \"kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.385668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.385569    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.385782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.385766    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.487055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.487014    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.487268 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.487137    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmtc\" (UniqueName: \"kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.487268 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.487223    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.487999 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.487973    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.489685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.489664    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.506314 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.506280    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmtc\" (UniqueName: \"kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc\") pod \"maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.626383 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.626287    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:22.954467 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:22.954439    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"]
Oct 30 22:09:22.956452 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:09:22.956421    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cfb020_31df_41d7_9299_e7bfdfb02e18.slice/crio-6af353f8f6acfb2f5f40f662c68822dee80b99a2c3c70ab5f54ca295c3ea7dc3 WatchSource:0}: Error finding container 6af353f8f6acfb2f5f40f662c68822dee80b99a2c3c70ab5f54ca295c3ea7dc3: Status 404 returned error can't find the container with id 6af353f8f6acfb2f5f40f662c68822dee80b99a2c3c70ab5f54ca295c3ea7dc3
Oct 30 22:09:23.912316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:23.912276    6695 generic.go:356] "Generic (PLEG): container finished" podID="b6cfb020-31df-41d7-9299-e7bfdfb02e18" containerID="788a8cef47c3087d9f78931f0503573e37502b28ff75e63ba8a49589999b1992" exitCode=0
Oct 30 22:09:23.912741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:23.912346    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2" event={"ID":"b6cfb020-31df-41d7-9299-e7bfdfb02e18","Type":"ContainerDied","Data":"788a8cef47c3087d9f78931f0503573e37502b28ff75e63ba8a49589999b1992"}
Oct 30 22:09:23.912741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:23.912399    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2" event={"ID":"b6cfb020-31df-41d7-9299-e7bfdfb02e18","Type":"ContainerStarted","Data":"6af353f8f6acfb2f5f40f662c68822dee80b99a2c3c70ab5f54ca295c3ea7dc3"}
Oct 30 22:09:25.124631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.124583    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:25.233404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.233316    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca\") pod \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") "
Oct 30 22:09:25.233555 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.233493    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc\" (UniqueName: \"kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc\") pod \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") "
Oct 30 22:09:25.233644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.233587    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca\") pod \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\" (UID: \"b6cfb020-31df-41d7-9299-e7bfdfb02e18\") "
Oct 30 22:09:25.233733 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.233708    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca" pod "b6cfb020-31df-41d7-9299-e7bfdfb02e18" (UID: "b6cfb020-31df-41d7-9299-e7bfdfb02e18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:25.234273 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.234251    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:25.235731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.235702    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca" pod "b6cfb020-31df-41d7-9299-e7bfdfb02e18" (UID: "b6cfb020-31df-41d7-9299-e7bfdfb02e18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:25.235855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.235778    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc" pod "b6cfb020-31df-41d7-9299-e7bfdfb02e18" (UID: "b6cfb020-31df-41d7-9299-e7bfdfb02e18"). InnerVolumeSpecName "kube-api-access-jbmtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:25.335993 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.335958    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmtc\" (UniqueName: \"kubernetes.io/projected/b6cfb020-31df-41d7-9299-e7bfdfb02e18-kube-api-access-jbmtc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:25.335993 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.335988    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfb020-31df-41d7-9299-e7bfdfb02e18-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:09:25.932061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.932026    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"
Oct 30 22:09:25.932061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.932047    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2" event={"ID":"b6cfb020-31df-41d7-9299-e7bfdfb02e18","Type":"ContainerDied","Data":"6af353f8f6acfb2f5f40f662c68822dee80b99a2c3c70ab5f54ca295c3ea7dc3"}
Oct 30 22:09:25.932308 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:25.932100    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6af353f8f6acfb2f5f40f662c68822dee80b99a2c3c70ab5f54ca295c3ea7dc3"
Oct 30 22:09:29.102897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:29.102862    6695 scope.go:122] "RemoveContainer" containerID="8ec9011ab0a53d6c9468210265114fd4d1be8ec3c4a1d1bfb0cea7969773d0da"
Oct 30 22:09:29.114493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:29.114464    6695 scope.go:122] "RemoveContainer" containerID="1bff3188029bae406f78316d7c14e339c30843508e741a444b4d0374514c68e1"
Oct 30 22:09:29.125175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:09:29.125152    6695 scope.go:122] "RemoveContainer" containerID="316beeea8e6e98cbc39fedcf5ae84c7b25b637d54a96a2733ad0b230866d704e"
Oct 30 22:09:31.078683 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:09:31.078636    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:10:01.031805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.031765    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"]
Oct 30 22:10:01.042583 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.042545    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4fb2eff-t6k8r"]
Oct 30 22:10:01.074806 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:01.074773    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:10:01.216289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.216252    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7556234a-e5de-4646-85c2-d97661cfa804" path="/var/lib/kubelet/pods/7556234a-e5de-4646-85c2-d97661cfa804/volumes"
Oct 30 22:10:01.901656 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.901596    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"]
Oct 30 22:10:01.902923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.902901    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b6cfb020-31df-41d7-9299-e7bfdfb02e18" containerName="revoke-keys"
Oct 30 22:10:01.903013 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.902924    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cfb020-31df-41d7-9299-e7bfdfb02e18" containerName="revoke-keys"
Oct 30 22:10:01.903156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.903144    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cfb020-31df-41d7-9299-e7bfdfb02e18" containerName="revoke-keys"
Oct 30 22:10:01.907429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.907405    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:01.921845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.921816    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:01.922177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.922157    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:10:01.925956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.925923    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"]
Oct 30 22:10:01.930970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.930951    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"]
Oct 30 22:10:01.931078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.931043    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:01.955019 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.954981    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"]
Oct 30 22:10:01.978528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.978485    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:01.978832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:01.978518    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:02.016672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.016578    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t46rw\" (UniqueName: \"kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.016898 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.016788    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.017110 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.017087    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.017210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.017140    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.017210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.017176    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.017341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.017218    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6l7\" (UniqueName: \"kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.119145 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.119104    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.119707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.119156    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.119707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.119318    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.119707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.119368    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6l7\" (UniqueName: \"kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.119707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.119424    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t46rw\" (UniqueName: \"kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.119707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.119553    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.120050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.120030    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.120436 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.120407    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.121666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.121643    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.121764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.121728    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.146022 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.145989    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6l7\" (UniqueName: \"kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7\") pod \"maas-api-revoke-sub-3347982f-q54dv\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.146152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.145991    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t46rw\" (UniqueName: \"kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw\") pod \"maas-api-revoke-sub-fa173de0-hldxw\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.229552 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.229467    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:02.273195 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.273157    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:02.544829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.544799    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"]
Oct 30 22:10:02.546747 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:10:02.546719    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf188c9f3_6f8b_41b0_950c_ddec78c802ea.slice/crio-31693ca02f8cdfcc40e023c34dd09abcd52b0e089ed1bba106de0fd1a835bb74 WatchSource:0}: Error finding container 31693ca02f8cdfcc40e023c34dd09abcd52b0e089ed1bba106de0fd1a835bb74: Status 404 returned error can't find the container with id 31693ca02f8cdfcc40e023c34dd09abcd52b0e089ed1bba106de0fd1a835bb74
Oct 30 22:10:02.605870 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:02.605840    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"]
Oct 30 22:10:02.608203 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:10:02.608174    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84c686f_3e63_4583_b56e_917e52c1db50.slice/crio-4a3e88150e0c4c607e3c231e29d0dfafdfc536ba77b88e22e73ebf4e5d4c0816 WatchSource:0}: Error finding container 4a3e88150e0c4c607e3c231e29d0dfafdfc536ba77b88e22e73ebf4e5d4c0816: Status 404 returned error can't find the container with id 4a3e88150e0c4c607e3c231e29d0dfafdfc536ba77b88e22e73ebf4e5d4c0816
Oct 30 22:10:03.260520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.260409    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv" event={"ID":"f84c686f-3e63-4583-b56e-917e52c1db50","Type":"ContainerStarted","Data":"2cade2b838a8d30d5b37aff06d2fe01e1207d47b9783d9ea7753bc46cd889484"}
Oct 30 22:10:03.260520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.260474    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv" event={"ID":"f84c686f-3e63-4583-b56e-917e52c1db50","Type":"ContainerStarted","Data":"4a3e88150e0c4c607e3c231e29d0dfafdfc536ba77b88e22e73ebf4e5d4c0816"}
Oct 30 22:10:03.262688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.262656    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw" event={"ID":"f188c9f3-6f8b-41b0-950c-ddec78c802ea","Type":"ContainerStarted","Data":"d355a062a11510616447bd0325416b329c60fac141d5e5f309ca5bd80b3c6a3b"}
Oct 30 22:10:03.262883 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.262725    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw" event={"ID":"f188c9f3-6f8b-41b0-950c-ddec78c802ea","Type":"ContainerStarted","Data":"31693ca02f8cdfcc40e023c34dd09abcd52b0e089ed1bba106de0fd1a835bb74"}
Oct 30 22:10:03.307205 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.307148    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv" podStartSLOduration=2.307133162 podStartE2EDuration="2.307133162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:03.303135424 +0000 UTC m=+3586.225593737" watchObservedRunningTime="2026-10-30 22:10:03.307133162 +0000 UTC m=+3586.229591454"
Oct 30 22:10:03.321018 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.320983    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"]
Oct 30 22:10:03.321241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.321196    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy" containerID="cri-o://cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099" gracePeriod=30
Oct 30 22:10:03.326354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.326318    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"]
Oct 30 22:10:03.326674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.326620    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy" containerID="cri-o://fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a" gracePeriod=30
Oct 30 22:10:03.333319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.332078    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"]
Oct 30 22:10:03.333319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.332282    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy" containerID="cri-o://db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e" gracePeriod=30
Oct 30 22:10:03.352044 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:03.351886    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw" podStartSLOduration=2.351866868 podStartE2EDuration="2.351866868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:03.347138155 +0000 UTC m=+3586.269596455" watchObservedRunningTime="2026-10-30 22:10:03.351866868 +0000 UTC m=+3586.274325168"
Oct 30 22:10:04.275937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:04.275901    6695 generic.go:356] "Generic (PLEG): container finished" podID="f188c9f3-6f8b-41b0-950c-ddec78c802ea" containerID="d355a062a11510616447bd0325416b329c60fac141d5e5f309ca5bd80b3c6a3b" exitCode=0
Oct 30 22:10:04.276445 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:04.275984    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw" event={"ID":"f188c9f3-6f8b-41b0-950c-ddec78c802ea","Type":"ContainerDied","Data":"d355a062a11510616447bd0325416b329c60fac141d5e5f309ca5bd80b3c6a3b"}
Oct 30 22:10:04.278384 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:04.278358    6695 generic.go:356] "Generic (PLEG): container finished" podID="f84c686f-3e63-4583-b56e-917e52c1db50" containerID="2cade2b838a8d30d5b37aff06d2fe01e1207d47b9783d9ea7753bc46cd889484" exitCode=0
Oct 30 22:10:04.278518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:04.278400    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv" event={"ID":"f84c686f-3e63-4583-b56e-917e52c1db50","Type":"ContainerDied","Data":"2cade2b838a8d30d5b37aff06d2fe01e1207d47b9783d9ea7753bc46cd889484"}
Oct 30 22:10:05.468869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.468818    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:05.513429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.513378    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:05.523203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.523171    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw\" (UniqueName: \"kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw\") pod \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") "
Oct 30 22:10:05.523375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.523356    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca\") pod \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") "
Oct 30 22:10:05.523775 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.523746    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca" pod "f188c9f3-6f8b-41b0-950c-ddec78c802ea" (UID: "f188c9f3-6f8b-41b0-950c-ddec78c802ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:05.524281 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.523841    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca\") pod \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\" (UID: \"f188c9f3-6f8b-41b0-950c-ddec78c802ea\") "
Oct 30 22:10:05.525484 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.525444    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:05.526347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.526320    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw" pod "f188c9f3-6f8b-41b0-950c-ddec78c802ea" (UID: "f188c9f3-6f8b-41b0-950c-ddec78c802ea"). InnerVolumeSpecName "kube-api-access-t46rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:05.526860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.526836    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca" pod "f188c9f3-6f8b-41b0-950c-ddec78c802ea" (UID: "f188c9f3-6f8b-41b0-950c-ddec78c802ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:05.626661 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.626596    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca\") pod \"f84c686f-3e63-4583-b56e-917e52c1db50\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") "
Oct 30 22:10:05.626842 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.626786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca\") pod \"f84c686f-3e63-4583-b56e-917e52c1db50\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") "
Oct 30 22:10:05.626941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.626922    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7\" (UniqueName: \"kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7\") pod \"f84c686f-3e63-4583-b56e-917e52c1db50\" (UID: \"f84c686f-3e63-4583-b56e-917e52c1db50\") "
Oct 30 22:10:05.626985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.626959    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca" pod "f84c686f-3e63-4583-b56e-917e52c1db50" (UID: "f84c686f-3e63-4583-b56e-917e52c1db50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:05.627575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.627552    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:05.627575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.627576    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t46rw\" (UniqueName: \"kubernetes.io/projected/f188c9f3-6f8b-41b0-950c-ddec78c802ea-kube-api-access-t46rw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:05.627821 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.627586    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f188c9f3-6f8b-41b0-950c-ddec78c802ea-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:05.628926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.628897    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca" pod "f84c686f-3e63-4583-b56e-917e52c1db50" (UID: "f84c686f-3e63-4583-b56e-917e52c1db50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:05.629010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.628961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7" pod "f84c686f-3e63-4583-b56e-917e52c1db50" (UID: "f84c686f-3e63-4583-b56e-917e52c1db50"). InnerVolumeSpecName "kube-api-access-7w6l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:05.729573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.729537    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84c686f-3e63-4583-b56e-917e52c1db50-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:05.729573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:05.729575    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w6l7\" (UniqueName: \"kubernetes.io/projected/f84c686f-3e63-4583-b56e-917e52c1db50-kube-api-access-7w6l7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:06.297986 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:06.297958    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"
Oct 30 22:10:06.298208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:06.297997    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv" event={"ID":"f84c686f-3e63-4583-b56e-917e52c1db50","Type":"ContainerDied","Data":"4a3e88150e0c4c607e3c231e29d0dfafdfc536ba77b88e22e73ebf4e5d4c0816"}
Oct 30 22:10:06.298208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:06.298051    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a3e88150e0c4c607e3c231e29d0dfafdfc536ba77b88e22e73ebf4e5d4c0816"
Oct 30 22:10:06.299731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:06.299706    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"
Oct 30 22:10:06.299846 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:06.299745    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw" event={"ID":"f188c9f3-6f8b-41b0-950c-ddec78c802ea","Type":"ContainerDied","Data":"31693ca02f8cdfcc40e023c34dd09abcd52b0e089ed1bba106de0fd1a835bb74"}
Oct 30 22:10:06.299846 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:06.299780    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31693ca02f8cdfcc40e023c34dd09abcd52b0e089ed1bba106de0fd1a835bb74"
Oct 30 22:10:08.680846 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.680818    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 22:10:08.684519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.684492    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 22:10:08.717172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.717144    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 22:10:08.800651 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800587    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.800860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800677    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.800860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800740    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.800860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800802    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.801045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800900    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.801045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800985    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.801045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.800990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.801045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801040    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.801275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801095    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.801275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801134    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.801275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801180    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.801275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801216    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.801275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801265    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.801554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801329    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.801554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801383    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.801554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801440    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.801554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801538    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.801815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801644    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.801815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801690    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.801815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801721    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.801815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801764    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.802116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801871    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.802116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.801930    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.802116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.802003    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs\") pod \"78e13172-cb33-402f-b875-5a1991357f13\" (UID: \"78e13172-cb33-402f-b875-5a1991357f13\") "
Oct 30 22:10:08.802116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.802048    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.802116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.802084    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket\") pod \"b4e053cb-05bf-4a0c-8749-0871878e6a35\" (UID: \"b4e053cb-05bf-4a0c-8749-0871878e6a35\") "
Oct 30 22:10:08.803200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.802120    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.803356 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.803249    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token\") pod \"94169247-0d0d-4d28-aa55-039baffc64ed\" (UID: \"94169247-0d0d-4d28-aa55-039baffc64ed\") "
Oct 30 22:10:08.803978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.803948    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "kube-api-access-5qg8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:08.804080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.803987    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.804414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.804393    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.804823 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.804806    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.805013 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.804998    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qg8b\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-kube-api-access-5qg8b\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.805346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.805309    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:08.805669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.805598    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.807490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.807462    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "kube-api-access-8drng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:08.807847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.807827    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.808006 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.807979    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:08.808259 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.808239    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.808364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.808308    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:08.809872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.809835    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.810180 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.810142    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:08.810424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.810398    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.810686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.810662    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.811014 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.810991    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:08.811245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.811218    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:08.811329 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.811266    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.811521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.811499    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.811638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.811581    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.811875 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.811852    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.812019 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.811990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:08.812290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.812265    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:08.812881 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.812859    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy" pod "b4e053cb-05bf-4a0c-8749-0871878e6a35" (UID: "b4e053cb-05bf-4a0c-8749-0871878e6a35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.816847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.816822    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.819409 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.819382    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:08.820159 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.820126    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx" pod "78e13172-cb33-402f-b875-5a1991357f13" (UID: "78e13172-cb33-402f-b875-5a1991357f13"). InnerVolumeSpecName "kube-api-access-g5vwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:08.820274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.820213    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy" pod "94169247-0d0d-4d28-aa55-039baffc64ed" (UID: "94169247-0d0d-4d28-aa55-039baffc64ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:08.906718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906638    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8drng\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-kube-api-access-8drng\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906704    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906720    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906734    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906748    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94169247-0d0d-4d28-aa55-039baffc64ed-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906762    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94169247-0d0d-4d28-aa55-039baffc64ed-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906776    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906790    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906803    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906817    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906830    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e13172-cb33-402f-b875-5a1991357f13-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906844    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906858    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906870    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906883    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.906904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906896    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906909    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e13172-cb33-402f-b875-5a1991357f13-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906923    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906936    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e053cb-05bf-4a0c-8749-0871878e6a35-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906948    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e13172-cb33-402f-b875-5a1991357f13-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906961    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94169247-0d0d-4d28-aa55-039baffc64ed-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906975    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e053cb-05bf-4a0c-8749-0871878e6a35-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.906987    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94169247-0d0d-4d28-aa55-039baffc64ed-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:08.907420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:08.907001    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5vwx\" (UniqueName: \"kubernetes.io/projected/78e13172-cb33-402f-b875-5a1991357f13-kube-api-access-g5vwx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:09.325258 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.325227    6695 generic.go:356] "Generic (PLEG): container finished" podID="78e13172-cb33-402f-b875-5a1991357f13" containerID="db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e" exitCode=0
Oct 30 22:10:09.325419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.325288    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"
Oct 30 22:10:09.325419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.325309    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" event={"ID":"78e13172-cb33-402f-b875-5a1991357f13","Type":"ContainerDied","Data":"db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e"}
Oct 30 22:10:09.325419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.325361    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm" event={"ID":"78e13172-cb33-402f-b875-5a1991357f13","Type":"ContainerDied","Data":"e215d677105c8e25a1dcc5235e3f179f55e19e8215d93b283e1626f326df70d3"}
Oct 30 22:10:09.325419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.325383    6695 scope.go:122] "RemoveContainer" containerID="db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e"
Oct 30 22:10:09.327131 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.327107    6695 generic.go:356] "Generic (PLEG): container finished" podID="94169247-0d0d-4d28-aa55-039baffc64ed" containerID="cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099" exitCode=0
Oct 30 22:10:09.327256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.327200    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"
Oct 30 22:10:09.327256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.327239    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" event={"ID":"94169247-0d0d-4d28-aa55-039baffc64ed","Type":"ContainerDied","Data":"cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099"}
Oct 30 22:10:09.327395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.327288    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9" event={"ID":"94169247-0d0d-4d28-aa55-039baffc64ed","Type":"ContainerDied","Data":"13d52b3d53ea48c17817358f30bce246ba096c8590a2db4ae876393dd6fc872c"}
Oct 30 22:10:09.329025 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.329001    6695 generic.go:356] "Generic (PLEG): container finished" podID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerID="fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a" exitCode=0
Oct 30 22:10:09.329129 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.329067    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"
Oct 30 22:10:09.329129 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.329073    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" event={"ID":"b4e053cb-05bf-4a0c-8749-0871878e6a35","Type":"ContainerDied","Data":"fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a"}
Oct 30 22:10:09.329129 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.329115    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm" event={"ID":"b4e053cb-05bf-4a0c-8749-0871878e6a35","Type":"ContainerDied","Data":"3e597d004893017d9e23c625e85968beeca48494c7786f4195c9bb8373331db6"}
Oct 30 22:10:09.338399 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.338383    6695 scope.go:122] "RemoveContainer" containerID="db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e"
Oct 30 22:10:09.338689 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:09.338670    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e\": container with ID starting with db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e not found: ID does not exist" containerID="db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e"
Oct 30 22:10:09.338752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.338695    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e"} err="failed to get container status \"db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e\": rpc error: code = NotFound desc = could not find container \"db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e\": container with ID starting with db30c84545ad4551c9038261c39c01d0631da0bc93a8b037b8997ebec87a645e not found: ID does not exist"
Oct 30 22:10:09.338752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.338712    6695 scope.go:122] "RemoveContainer" containerID="cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099"
Oct 30 22:10:09.349108 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.349035    6695 scope.go:122] "RemoveContainer" containerID="cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099"
Oct 30 22:10:09.349649 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:09.349614    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099\": container with ID starting with cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099 not found: ID does not exist" containerID="cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099"
Oct 30 22:10:09.349752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.349648    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099"} err="failed to get container status \"cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099\": rpc error: code = NotFound desc = could not find container \"cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099\": container with ID starting with cce1eff4e3c4b71f4a8d5101f296a5baf7234753278e5c6719230750d8991099 not found: ID does not exist"
Oct 30 22:10:09.349752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.349670    6695 scope.go:122] "RemoveContainer" containerID="fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a"
Oct 30 22:10:09.382418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.382385    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"]
Oct 30 22:10:09.387105 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.387086    6695 scope.go:122] "RemoveContainer" containerID="fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a"
Oct 30 22:10:09.387589 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:09.387565    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a\": container with ID starting with fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a not found: ID does not exist" containerID="fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a"
Oct 30 22:10:09.387716 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.387598    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a"} err="failed to get container status \"fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a\": rpc error: code = NotFound desc = could not find container \"fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a\": container with ID starting with fd46a76f1b00996652e3abf9d87918155685388a99ba948d629f31c12853b50a not found: ID does not exist"
Oct 30 22:10:09.393089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.393060    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fj2hm"]
Oct 30 22:10:09.429980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.429945    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"]
Oct 30 22:10:09.434651 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.434625    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-j6jq9"]
Oct 30 22:10:09.467872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.467842    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"]
Oct 30 22:10:09.473173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:09.473144    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-rc9jm"]
Oct 30 22:10:11.036287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:11.036250    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"]
Oct 30 22:10:11.046437 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:11.046409    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcfbb4f7-js4l7"]
Oct 30 22:10:11.226684 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:11.226639    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005a4540-df13-41cb-8371-3820f879b8bf" path="/var/lib/kubelet/pods/005a4540-df13-41cb-8371-3820f879b8bf/volumes"
Oct 30 22:10:11.227760 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:11.227733    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e13172-cb33-402f-b875-5a1991357f13" path="/var/lib/kubelet/pods/78e13172-cb33-402f-b875-5a1991357f13/volumes"
Oct 30 22:10:11.228621 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:11.228580    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" path="/var/lib/kubelet/pods/94169247-0d0d-4d28-aa55-039baffc64ed/volumes"
Oct 30 22:10:11.229761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:11.229734    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" path="/var/lib/kubelet/pods/b4e053cb-05bf-4a0c-8749-0871878e6a35/volumes"
Oct 30 22:10:20.765392 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:20.765361    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 22:10:20.801269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:20.801243    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 22:10:22.592359 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.592320    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"]
Oct 30 22:10:22.593153 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593135    6695 cpu_manager.go:414] "containerMap: removing container" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy"
Oct 30 22:10:22.593203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593153    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy"
Oct 30 22:10:22.593203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593171    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f84c686f-3e63-4583-b56e-917e52c1db50" containerName="revoke-keys"
Oct 30 22:10:22.593203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593178    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84c686f-3e63-4583-b56e-917e52c1db50" containerName="revoke-keys"
Oct 30 22:10:22.593315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593202    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy"
Oct 30 22:10:22.593315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593208    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy"
Oct 30 22:10:22.593315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593235    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f188c9f3-6f8b-41b0-950c-ddec78c802ea" containerName="revoke-keys"
Oct 30 22:10:22.593315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593239    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f188c9f3-6f8b-41b0-950c-ddec78c802ea" containerName="revoke-keys"
Oct 30 22:10:22.593315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593252    6695 cpu_manager.go:414] "containerMap: removing container" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy"
Oct 30 22:10:22.593315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593256    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy"
Oct 30 22:10:22.593515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593364    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f188c9f3-6f8b-41b0-950c-ddec78c802ea" containerName="revoke-keys"
Oct 30 22:10:22.593515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593376    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e13172-cb33-402f-b875-5a1991357f13" containerName="istio-proxy"
Oct 30 22:10:22.593515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593386    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="94169247-0d0d-4d28-aa55-039baffc64ed" containerName="istio-proxy"
Oct 30 22:10:22.593515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593394    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84c686f-3e63-4583-b56e-917e52c1db50" containerName="revoke-keys"
Oct 30 22:10:22.593515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.593403    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e053cb-05bf-4a0c-8749-0871878e6a35" containerName="istio-proxy"
Oct 30 22:10:22.597157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.597138    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.621411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.621382    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"]
Oct 30 22:10:22.626298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.626278    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:22.626298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.626288    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:22.648485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.648452    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:22.649031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.649007    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:10:22.708988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.708948    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.709176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.709110    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sds5s\" (UniqueName: \"kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.709176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.709170    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.810424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.810383    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.810634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.810497    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sds5s\" (UniqueName: \"kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.810634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.810555    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.811292 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.811270    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.812735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.812712    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.832367 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.832340    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sds5s\" (UniqueName: \"kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s\") pod \"maas-api-revoke-sub-960ad0ba-rknpl\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:22.911830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:22.911744    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:23.224133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:23.224096    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"]
Oct 30 22:10:23.224783 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:10:23.224753    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23cab2c_9e7f_45ce_ae1b_263fce203269.slice/crio-8b30485063eb2775a33f5321a595295c2c5470bd7d9884e2ded8c75276d7db6f WatchSource:0}: Error finding container 8b30485063eb2775a33f5321a595295c2c5470bd7d9884e2ded8c75276d7db6f: Status 404 returned error can't find the container with id 8b30485063eb2775a33f5321a595295c2c5470bd7d9884e2ded8c75276d7db6f
Oct 30 22:10:23.461247 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:23.461214    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl" event={"ID":"c23cab2c-9e7f-45ce-ae1b-263fce203269","Type":"ContainerStarted","Data":"8b30485063eb2775a33f5321a595295c2c5470bd7d9884e2ded8c75276d7db6f"}
Oct 30 22:10:24.471672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:24.471573    6695 generic.go:356] "Generic (PLEG): container finished" podID="c23cab2c-9e7f-45ce-ae1b-263fce203269" containerID="cbdcfd0438b22a9957bb12221b194b4fe82f2fc4a5430ef0ee9f3b7c366bc008" exitCode=0
Oct 30 22:10:24.472057 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:24.471665    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl" event={"ID":"c23cab2c-9e7f-45ce-ae1b-263fce203269","Type":"ContainerDied","Data":"cbdcfd0438b22a9957bb12221b194b4fe82f2fc4a5430ef0ee9f3b7c366bc008"}
Oct 30 22:10:25.657148 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.657122    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:25.763438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.763360    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca\") pod \"c23cab2c-9e7f-45ce-ae1b-263fce203269\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") "
Oct 30 22:10:25.763438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.763405    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca\") pod \"c23cab2c-9e7f-45ce-ae1b-263fce203269\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") "
Oct 30 22:10:25.763759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.763743    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s\" (UniqueName: \"kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s\") pod \"c23cab2c-9e7f-45ce-ae1b-263fce203269\" (UID: \"c23cab2c-9e7f-45ce-ae1b-263fce203269\") "
Oct 30 22:10:25.763809 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.763782    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca" pod "c23cab2c-9e7f-45ce-ae1b-263fce203269" (UID: "c23cab2c-9e7f-45ce-ae1b-263fce203269"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:25.764311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.764294    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:25.765501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.765476    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca" pod "c23cab2c-9e7f-45ce-ae1b-263fce203269" (UID: "c23cab2c-9e7f-45ce-ae1b-263fce203269"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:25.765819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.765794    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s" pod "c23cab2c-9e7f-45ce-ae1b-263fce203269" (UID: "c23cab2c-9e7f-45ce-ae1b-263fce203269"). InnerVolumeSpecName "kube-api-access-sds5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:25.865970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.865827    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sds5s\" (UniqueName: \"kubernetes.io/projected/c23cab2c-9e7f-45ce-ae1b-263fce203269-kube-api-access-sds5s\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:25.865970 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:25.865863    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23cab2c-9e7f-45ce-ae1b-263fce203269-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:26.490310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:26.490277    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"
Oct 30 22:10:26.490487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:26.490323    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl" event={"ID":"c23cab2c-9e7f-45ce-ae1b-263fce203269","Type":"ContainerDied","Data":"8b30485063eb2775a33f5321a595295c2c5470bd7d9884e2ded8c75276d7db6f"}
Oct 30 22:10:26.490487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:26.490418    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b30485063eb2775a33f5321a595295c2c5470bd7d9884e2ded8c75276d7db6f"
Oct 30 22:10:29.200540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:29.200456    6695 scope.go:122] "RemoveContainer" containerID="c1a8b695b3a9568eb11413687b41455028997ade6eb07193c333134036805211"
Oct 30 22:10:29.213245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:29.213216    6695 scope.go:122] "RemoveContainer" containerID="0aed215f8573243807e174a46572f308bf2dd6eb4822ceb055fa126c9353fd16"
Oct 30 22:10:29.223332 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:29.223312    6695 scope.go:122] "RemoveContainer" containerID="72311d1e86ab46765b1e66f2934be8c2e2acbc8918e8e7586c517ea2980414d1"
Oct 30 22:10:31.074170 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:31.074136    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:10:33.441599 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.441561    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"]
Oct 30 22:10:33.946882 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.946847    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"]
Oct 30 22:10:33.947674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.947659    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c23cab2c-9e7f-45ce-ae1b-263fce203269" containerName="revoke-keys"
Oct 30 22:10:33.947729 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.947675    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23cab2c-9e7f-45ce-ae1b-263fce203269" containerName="revoke-keys"
Oct 30 22:10:33.947816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.947807    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23cab2c-9e7f-45ce-ae1b-263fce203269" containerName="revoke-keys"
Oct 30 22:10:33.953812 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.953795    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:33.969759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.969728    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"]
Oct 30 22:10:33.976478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.976450    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:33.976721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.976497    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:33.976721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.976521    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:33.976903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:33.976534    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:10:34.023463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.023430    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.023648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.023615    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.023740 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.023728    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6mx\" (UniqueName: \"kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.125080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.125044    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6mx\" (UniqueName: \"kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.125295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.125182    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.125361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.125317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.125900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.125877    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.127808 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.127780    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.144556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.144530    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6mx\" (UniqueName: \"kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx\") pod \"maas-api-revoke-sub-7061b7d6-75k7r\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.274240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.274213    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:34.581453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.581415    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"]
Oct 30 22:10:34.583712 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:10:34.583681    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6c1fab_882b_4f6d_aae9_72d5d96e7ec8.slice/crio-f3401e26cee79380bc1f9a67c706f39a5ec96ccea601eef37695410e85e42296 WatchSource:0}: Error finding container f3401e26cee79380bc1f9a67c706f39a5ec96ccea601eef37695410e85e42296: Status 404 returned error can't find the container with id f3401e26cee79380bc1f9a67c706f39a5ec96ccea601eef37695410e85e42296
Oct 30 22:10:34.642555 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:34.642514    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r" event={"ID":"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8","Type":"ContainerStarted","Data":"f3401e26cee79380bc1f9a67c706f39a5ec96ccea601eef37695410e85e42296"}
Oct 30 22:10:35.652987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:35.652950    6695 generic.go:356] "Generic (PLEG): container finished" podID="da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" containerID="a29e3665c5f2f86d8675663a7975932069329ade5f88d2b95b1d9af48f30375a" exitCode=0
Oct 30 22:10:35.653411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:35.653049    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r" event={"ID":"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8","Type":"ContainerDied","Data":"a29e3665c5f2f86d8675663a7975932069329ade5f88d2b95b1d9af48f30375a"}
Oct 30 22:10:36.832340 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.832310    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:36.968907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.968823    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx\" (UniqueName: \"kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx\") pod \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") "
Oct 30 22:10:36.969050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.968969    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca\") pod \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") "
Oct 30 22:10:36.969050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.968990    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca\") pod \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\" (UID: \"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8\") "
Oct 30 22:10:36.969356 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.969320    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca" pod "da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" (UID: "da6c1fab-882b-4f6d-aae9-72d5d96e7ec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:36.969641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.969623    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:36.970961 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.970942    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx" pod "da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" (UID: "da6c1fab-882b-4f6d-aae9-72d5d96e7ec8"). InnerVolumeSpecName "kube-api-access-9t6mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:36.971074 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:36.971053    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca" pod "da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" (UID: "da6c1fab-882b-4f6d-aae9-72d5d96e7ec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:37.071058 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:37.071011    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6mx\" (UniqueName: \"kubernetes.io/projected/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-kube-api-access-9t6mx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:37.071058 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:37.071051    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:37.673640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:37.673586    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"
Oct 30 22:10:37.673640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:37.673593    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r" event={"ID":"da6c1fab-882b-4f6d-aae9-72d5d96e7ec8","Type":"ContainerDied","Data":"f3401e26cee79380bc1f9a67c706f39a5ec96ccea601eef37695410e85e42296"}
Oct 30 22:10:37.673916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:37.673681    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3401e26cee79380bc1f9a67c706f39a5ec96ccea601eef37695410e85e42296"
Oct 30 22:10:45.558624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:45.556326    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"]
Oct 30 22:10:45.568596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:45.568562    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-50ab77-hwjg2"]
Oct 30 22:10:47.216189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:47.216149    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cfb020-31df-41d7-9299-e7bfdfb02e18" path="/var/lib/kubelet/pods/b6cfb020-31df-41d7-9299-e7bfdfb02e18/volumes"
Oct 30 22:10:48.463148 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.463103    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" podUID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" containerName="main" containerID="cri-o://b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb" gracePeriod=45
Oct 30 22:10:48.742504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.742475    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:10:48.824193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.824151    6695 generic.go:356] "Generic (PLEG): container finished" podID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" containerID="b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb" exitCode=0
Oct 30 22:10:48.824372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.824260    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"
Oct 30 22:10:48.824372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.824286    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" event={"ID":"bdeddcd0-d866-4ebc-867b-f5184af92bb1","Type":"ContainerDied","Data":"b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb"}
Oct 30 22:10:48.824372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.824341    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" event={"ID":"bdeddcd0-d866-4ebc-867b-f5184af92bb1","Type":"ContainerDied","Data":"0f424ccc7bce6a2a917dfcef46d3f6c95488ebd78d56cf77d072e43d2abe0f2c"}
Oct 30 22:10:48.824372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.824365    6695 scope.go:122] "RemoveContainer" containerID="b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb"
Oct 30 22:10:48.838111 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.838076    6695 scope.go:122] "RemoveContainer" containerID="b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb"
Oct 30 22:10:48.838466 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:48.838432    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb\": container with ID starting with b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb not found: ID does not exist" containerID="b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb"
Oct 30 22:10:48.838566 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.838479    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb"} err="failed to get container status \"b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb\": rpc error: code = NotFound desc = could not find container \"b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb\": container with ID starting with b2b31da3d7c3648b4705f9489c7ebda4fd8b41217ca1914e842f935776d0d2fb not found: ID does not exist"
Oct 30 22:10:48.924445 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924407    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm\") pod \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") "
Oct 30 22:10:48.924627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924549    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms\" (UniqueName: \"kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms\") pod \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") "
Oct 30 22:10:48.924627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924597    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir\") pod \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") "
Oct 30 22:10:48.924762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924654    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home\") pod \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") "
Oct 30 22:10:48.924762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924694    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs\") pod \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") "
Oct 30 22:10:48.924892 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924767    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache\") pod \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\" (UID: \"bdeddcd0-d866-4ebc-867b-f5184af92bb1\") "
Oct 30 22:10:48.924977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924936    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir" pod "bdeddcd0-d866-4ebc-867b-f5184af92bb1" (UID: "bdeddcd0-d866-4ebc-867b-f5184af92bb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.925095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.924982    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home" pod "bdeddcd0-d866-4ebc-867b-f5184af92bb1" (UID: "bdeddcd0-d866-4ebc-867b-f5184af92bb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.925269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.925245    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache" pod "bdeddcd0-d866-4ebc-867b-f5184af92bb1" (UID: "bdeddcd0-d866-4ebc-867b-f5184af92bb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.925539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.925518    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:48.925597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.925547    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:48.925597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.925561    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:48.927013 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.926978    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs" pod "bdeddcd0-d866-4ebc-867b-f5184af92bb1" (UID: "bdeddcd0-d866-4ebc-867b-f5184af92bb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:48.927181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.927160    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm" pod "bdeddcd0-d866-4ebc-867b-f5184af92bb1" (UID: "bdeddcd0-d866-4ebc-867b-f5184af92bb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.927265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:48.927203    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms" pod "bdeddcd0-d866-4ebc-867b-f5184af92bb1" (UID: "bdeddcd0-d866-4ebc-867b-f5184af92bb1"). InnerVolumeSpecName "kube-api-access-sg7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:49.027208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.027170    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdeddcd0-d866-4ebc-867b-f5184af92bb1-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:49.027208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.027198    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdeddcd0-d866-4ebc-867b-f5184af92bb1-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:49.027208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.027207    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7ms\" (UniqueName: \"kubernetes.io/projected/bdeddcd0-d866-4ebc-867b-f5184af92bb1-kube-api-access-sg7ms\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:49.172788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.172758    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"]
Oct 30 22:10:49.182165 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.182133    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4"]
Oct 30 22:10:49.244732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.239686    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" path="/var/lib/kubelet/pods/bdeddcd0-d866-4ebc-867b-f5184af92bb1/volumes"
Oct 30 22:10:49.711861 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:49.711816    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8d53e0/test-model-8d53e0-kserve-7d9dbbc765-xmzq4" podUID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" containerName="main" probeResult="failure" output="Get \"https://10.134.1.199:8000/ready\": context deadline exceeded"
Oct 30 22:10:51.519172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.519138    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"]
Oct 30 22:10:51.772071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.771992    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"]
Oct 30 22:10:51.772836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.772817    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" containerName="main"
Oct 30 22:10:51.772836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.772835    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" containerName="main"
Oct 30 22:10:51.772980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.772881    6695 cpu_manager.go:414] "containerMap: removing container" podUID="da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" containerName="revoke-keys"
Oct 30 22:10:51.772980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.772887    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" containerName="revoke-keys"
Oct 30 22:10:51.773051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.773019    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdeddcd0-d866-4ebc-867b-f5184af92bb1" containerName="main"
Oct 30 22:10:51.773051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.773034    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" containerName="revoke-keys"
Oct 30 22:10:51.777871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.777849    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.792771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.792744    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:51.792949 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.792903    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:10:51.794844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.794821    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"]
Oct 30 22:10:51.845707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.845677    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:51.845707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.845688    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:51.879731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.879696    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.879931 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.879909    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.880002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.879980    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b67t9\" (UniqueName: \"kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.981503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.981466    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.981738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.981537    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67t9\" (UniqueName: \"kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.981738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.981643    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.982311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.982282    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:51.983913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:51.983893    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:52.001897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.001871    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67t9\" (UniqueName: \"kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9\") pod \"maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:52.092955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.092873    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:52.120379 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.120341    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"]
Oct 30 22:10:52.120648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.120593    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy" containerID="cri-o://8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5" gracePeriod=30
Oct 30 22:10:52.419486 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.419458    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"]
Oct 30 22:10:52.421849 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:10:52.421813    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08926c1_5562_49b8_8ad7_c297592a805b.slice/crio-9c82890070235b9a6cc423d711026aef3d6be424db2727d3de684a5c8a5b0c50 WatchSource:0}: Error finding container 9c82890070235b9a6cc423d711026aef3d6be424db2727d3de684a5c8a5b0c50: Status 404 returned error can't find the container with id 9c82890070235b9a6cc423d711026aef3d6be424db2727d3de684a5c8a5b0c50
Oct 30 22:10:52.922052 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.922009    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg" event={"ID":"b08926c1-5562-49b8-8ad7-c297592a805b","Type":"ContainerStarted","Data":"a3d91033647335b1de188a96f8054da6899912eaa5c1e2716b5b40f62ff57bc2"}
Oct 30 22:10:52.922654 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.922078    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg" event={"ID":"b08926c1-5562-49b8-8ad7-c297592a805b","Type":"ContainerStarted","Data":"9c82890070235b9a6cc423d711026aef3d6be424db2727d3de684a5c8a5b0c50"}
Oct 30 22:10:52.941274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.941238    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"]
Oct 30 22:10:52.941458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.941437    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy" containerID="cri-o://c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934" gracePeriod=30
Oct 30 22:10:52.978084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:52.977958    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg" podStartSLOduration=1.977937909 podStartE2EDuration="1.977937909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:52.975103763 +0000 UTC m=+3635.897562073" watchObservedRunningTime="2026-10-30 22:10:52.977937909 +0000 UTC m=+3635.900396207"
Oct 30 22:10:53.931629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:53.931569    6695 generic.go:356] "Generic (PLEG): container finished" podID="b08926c1-5562-49b8-8ad7-c297592a805b" containerID="a3d91033647335b1de188a96f8054da6899912eaa5c1e2716b5b40f62ff57bc2" exitCode=0
Oct 30 22:10:53.932073 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:53.931633    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg" event={"ID":"b08926c1-5562-49b8-8ad7-c297592a805b","Type":"ContainerDied","Data":"a3d91033647335b1de188a96f8054da6899912eaa5c1e2716b5b40f62ff57bc2"}
Oct 30 22:10:55.111415 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.111376    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:55.150320 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.150291    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9\" (UniqueName: \"kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9\") pod \"b08926c1-5562-49b8-8ad7-c297592a805b\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") "
Oct 30 22:10:55.150492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.150471    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca\") pod \"b08926c1-5562-49b8-8ad7-c297592a805b\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") "
Oct 30 22:10:55.150676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.150659    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca\") pod \"b08926c1-5562-49b8-8ad7-c297592a805b\" (UID: \"b08926c1-5562-49b8-8ad7-c297592a805b\") "
Oct 30 22:10:55.151065 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.150990    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca" pod "b08926c1-5562-49b8-8ad7-c297592a805b" (UID: "b08926c1-5562-49b8-8ad7-c297592a805b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:55.151298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.151280    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:55.152518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.152496    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca" pod "b08926c1-5562-49b8-8ad7-c297592a805b" (UID: "b08926c1-5562-49b8-8ad7-c297592a805b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:55.152632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.152560    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9" pod "b08926c1-5562-49b8-8ad7-c297592a805b" (UID: "b08926c1-5562-49b8-8ad7-c297592a805b"). InnerVolumeSpecName "kube-api-access-b67t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:55.252182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.252147    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b08926c1-5562-49b8-8ad7-c297592a805b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:55.252182 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.252181    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b67t9\" (UniqueName: \"kubernetes.io/projected/b08926c1-5562-49b8-8ad7-c297592a805b-kube-api-access-b67t9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:55.948863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.948821    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"
Oct 30 22:10:55.949050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.948862    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg" event={"ID":"b08926c1-5562-49b8-8ad7-c297592a805b","Type":"ContainerDied","Data":"9c82890070235b9a6cc423d711026aef3d6be424db2727d3de684a5c8a5b0c50"}
Oct 30 22:10:55.949050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:55.948941    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c82890070235b9a6cc423d711026aef3d6be424db2727d3de684a5c8a5b0c50"
Oct 30 22:10:57.408301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.408274    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 22:10:57.485088 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485054    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485111    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485172    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485265    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485356    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:57.485472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485432    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485473    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485540    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:57.485595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485559    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:57.485595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485588    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485639    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485737    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert\") pod \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\" (UID: \"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe\") "
Oct 30 22:10:57.485826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.485747    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:57.486185 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.486157    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:57.486558 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.486531    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.486694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.486559    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.486694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.486574    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.486694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.486587    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.486694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.486630    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.487865 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.487836    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:57.488149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.487961    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:57.488149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.487972    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "kube-api-access-nbggj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:57.488364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.488338    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo" pod "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" (UID: "5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:57.588183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.588148    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.588183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.588179    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.588183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.588189    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbggj\" (UniqueName: \"kubernetes.io/projected/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-kube-api-access-nbggj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.588463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.588198    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:57.966472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.966442    6695 generic.go:356] "Generic (PLEG): container finished" podID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerID="8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5" exitCode=0
Oct 30 22:10:57.966644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.966485    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"
Oct 30 22:10:57.966644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.966521    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" event={"ID":"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe","Type":"ContainerDied","Data":"8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5"}
Oct 30 22:10:57.966644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.966586    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8" event={"ID":"5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe","Type":"ContainerDied","Data":"3d256396b1bdec4c517fe388dee57cb0f521b318df114ac575d075a096960710"}
Oct 30 22:10:57.966644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:57.966632    6695 scope.go:122] "RemoveContainer" containerID="8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5"
Oct 30 22:10:58.057436 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.057322    6695 scope.go:122] "RemoveContainer" containerID="8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5"
Oct 30 22:10:58.057979 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:58.057899    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5\": container with ID starting with 8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5 not found: ID does not exist" containerID="8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5"
Oct 30 22:10:58.057979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.057939    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5"} err="failed to get container status \"8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5\": rpc error: code = NotFound desc = could not find container \"8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5\": container with ID starting with 8b5d608b8c4603d1b077a9435e8cb40c6fe8f210a6c81fd7bc67ab1044ac0cb5 not found: ID does not exist"
Oct 30 22:10:58.070134 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.070102    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"]
Oct 30 22:10:58.084768 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.084733    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5d3601-openshift-default-846c447f89-vq9r8"]
Oct 30 22:10:58.260143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.260116    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:10:58.298462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.298423    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.298679 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.298594    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.298762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.298687    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.298902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.298871    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.299141 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.299120    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.299336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.299322    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.299381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.299339    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.300641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.300620    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.413390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.413139    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.413390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.413199    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.413390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.413261    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.413390 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.413298    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.414140 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.413647    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.414140 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.413749    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert\") pod \"5675ebbf-042a-4a91-ab8c-1adacaf45777\" (UID: \"5675ebbf-042a-4a91-ab8c-1adacaf45777\") "
Oct 30 22:10:58.414502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.414470    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.424653 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.417848    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:58.424653 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.420693    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.425962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.425449    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.430568 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.429829    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "kube-api-access-8b6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:58.430982 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.430958    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:58.431757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.431716    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo" pod "5675ebbf-042a-4a91-ab8c-1adacaf45777" (UID: "5675ebbf-042a-4a91-ab8c-1adacaf45777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:58.516404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.516322    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.516404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.516358    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5675ebbf-042a-4a91-ab8c-1adacaf45777-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.516404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.516367    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.516404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.516376    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6kd\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-kube-api-access-8b6kd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.516404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.516387    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5675ebbf-042a-4a91-ab8c-1adacaf45777-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.516404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.516396    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5675ebbf-042a-4a91-ab8c-1adacaf45777-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.975908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.975873    6695 generic.go:356] "Generic (PLEG): container finished" podID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerID="c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934" exitCode=0
Oct 30 22:10:58.976126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.975931    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"
Oct 30 22:10:58.976126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.975959    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" event={"ID":"5675ebbf-042a-4a91-ab8c-1adacaf45777","Type":"ContainerDied","Data":"c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934"}
Oct 30 22:10:58.976126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.976019    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb" event={"ID":"5675ebbf-042a-4a91-ab8c-1adacaf45777","Type":"ContainerDied","Data":"e324469d58df23f99908a4fa9afa2451276fe41be927941ee6b13216229c7178"}
Oct 30 22:10:58.976126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.976042    6695 scope.go:122] "RemoveContainer" containerID="c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934"
Oct 30 22:10:58.987992 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.987975    6695 scope.go:122] "RemoveContainer" containerID="c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934"
Oct 30 22:10:58.988297 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:10:58.988278    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934\": container with ID starting with c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934 not found: ID does not exist" containerID="c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934"
Oct 30 22:10:58.988348 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:58.988306    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934"} err="failed to get container status \"c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934\": rpc error: code = NotFound desc = could not find container \"c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934\": container with ID starting with c05fca46537b13a3dc987d6aa533d8abd701d26afa02c2424e55ef2cec3fd934 not found: ID does not exist"
Oct 30 22:10:59.044991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:59.044954    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"]
Oct 30 22:10:59.056183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:59.056152    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-qq6lb"]
Oct 30 22:10:59.215309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:59.215276    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" path="/var/lib/kubelet/pods/5675ebbf-042a-4a91-ab8c-1adacaf45777/volumes"
Oct 30 22:10:59.215837 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:10:59.215823    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" path="/var/lib/kubelet/pods/5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe/volumes"
Oct 30 22:11:01.075705 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:01.075660    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:11:02.320500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:02.320446    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"]
Oct 30 22:11:02.356489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:02.356450    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"]
Oct 30 22:11:03.615790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:03.615756    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"]
Oct 30 22:11:03.616323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:03.615981    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy" containerID="cri-o://2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d" gracePeriod=30
Oct 30 22:11:06.540947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.540906    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl" podUID="0f0c618f-5bd8-4144-a480-f1c40234ce00" containerName="main" containerID="cri-o://ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164" gracePeriod=45
Oct 30 22:11:06.826370 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.826338    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:11:06.858794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.858754    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs\") pod \"0f0c618f-5bd8-4144-a480-f1c40234ce00\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") "
Oct 30 22:11:06.858980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.858834    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w\" (UniqueName: \"kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w\") pod \"0f0c618f-5bd8-4144-a480-f1c40234ce00\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") "
Oct 30 22:11:06.858980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.858951    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache\") pod \"0f0c618f-5bd8-4144-a480-f1c40234ce00\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") "
Oct 30 22:11:06.859177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.859160    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home\") pod \"0f0c618f-5bd8-4144-a480-f1c40234ce00\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") "
Oct 30 22:11:06.859236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.859206    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir\") pod \"0f0c618f-5bd8-4144-a480-f1c40234ce00\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") "
Oct 30 22:11:06.859296 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.859237    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm\") pod \"0f0c618f-5bd8-4144-a480-f1c40234ce00\" (UID: \"0f0c618f-5bd8-4144-a480-f1c40234ce00\") "
Oct 30 22:11:06.859444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.859396    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache" pod "0f0c618f-5bd8-4144-a480-f1c40234ce00" (UID: "0f0c618f-5bd8-4144-a480-f1c40234ce00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:06.859630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.859506    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home" pod "0f0c618f-5bd8-4144-a480-f1c40234ce00" (UID: "0f0c618f-5bd8-4144-a480-f1c40234ce00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:06.859730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.859661    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir" pod "0f0c618f-5bd8-4144-a480-f1c40234ce00" (UID: "0f0c618f-5bd8-4144-a480-f1c40234ce00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:06.860067 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.860042    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:06.860203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.860070    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:06.860203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.860084    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:06.861782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.861754    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm" pod "0f0c618f-5bd8-4144-a480-f1c40234ce00" (UID: "0f0c618f-5bd8-4144-a480-f1c40234ce00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:06.861782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.861761    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs" pod "0f0c618f-5bd8-4144-a480-f1c40234ce00" (UID: "0f0c618f-5bd8-4144-a480-f1c40234ce00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:06.861973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.861950    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w" pod "0f0c618f-5bd8-4144-a480-f1c40234ce00" (UID: "0f0c618f-5bd8-4144-a480-f1c40234ce00"). InnerVolumeSpecName "kube-api-access-pbf2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:06.961595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.961558    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0c618f-5bd8-4144-a480-f1c40234ce00-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:06.961595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.961593    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbf2w\" (UniqueName: \"kubernetes.io/projected/0f0c618f-5bd8-4144-a480-f1c40234ce00-kube-api-access-pbf2w\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:06.961595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:06.961621    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0c618f-5bd8-4144-a480-f1c40234ce00-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.076562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.076471    6695 generic.go:356] "Generic (PLEG): container finished" podID="0f0c618f-5bd8-4144-a480-f1c40234ce00" containerID="ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164" exitCode=0
Oct 30 22:11:07.076562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.076517    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"
Oct 30 22:11:07.076562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.076549    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl" event={"ID":"0f0c618f-5bd8-4144-a480-f1c40234ce00","Type":"ContainerDied","Data":"ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164"}
Oct 30 22:11:07.076838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.076591    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl" event={"ID":"0f0c618f-5bd8-4144-a480-f1c40234ce00","Type":"ContainerDied","Data":"df5e31abb45f84eba84f701277c768d9d31349f29d62e81312db744fca612842"}
Oct 30 22:11:07.076838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.076638    6695 scope.go:122] "RemoveContainer" containerID="ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164"
Oct 30 22:11:07.088581 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.088563    6695 scope.go:122] "RemoveContainer" containerID="ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164"
Oct 30 22:11:07.088839 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:07.088820    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164\": container with ID starting with ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164 not found: ID does not exist" containerID="ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164"
Oct 30 22:11:07.088989 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.088846    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164"} err="failed to get container status \"ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164\": rpc error: code = NotFound desc = could not find container \"ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164\": container with ID starting with ab80fd9e809eb5c4c0180ccb3d01960da5cedcc97a252dbd0351aedf112d0164 not found: ID does not exist"
Oct 30 22:11:07.139979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.139945    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"]
Oct 30 22:11:07.148804 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.148773    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6537e6/test-model-6537e6-kserve-57f5bf4d67-n9stl"]
Oct 30 22:11:07.219577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.219538    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0c618f-5bd8-4144-a480-f1c40234ce00" path="/var/lib/kubelet/pods/0f0c618f-5bd8-4144-a480-f1c40234ce00/volumes"
Oct 30 22:11:07.321119 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.321076    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerName="payload-processing" containerID="cri-o://2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5" gracePeriod=25
Oct 30 22:11:07.357811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:07.357704    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" containerName="payload-pre-processing" containerID="cri-o://3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489" gracePeriod=25
Oct 30 22:11:08.893544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.893477    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:11:08.988535 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988496    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.988535 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988539    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.988813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988570    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.988813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988728    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.988813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988794    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.989011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988824    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.989011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988841    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:08.989011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988860    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.989011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.988936    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:08.989346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.989306    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.989453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.989307    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:08.989453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.989362    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:08.989589 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.989500    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert\") pod \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\" (UID: \"ee7cc97d-228d-4f75-a3b9-914cf91dac07\") "
Oct 30 22:11:08.989984 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.989957    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:08.990594 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.990568    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.990751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.990634    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.990751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.990662    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.990751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.990686    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.990751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.990699    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:08.992347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.992320    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:08.992440 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.992363    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:08.992440 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.992399    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "kube-api-access-bztcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:08.992817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:08.992791    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo" pod "ee7cc97d-228d-4f75-a3b9-914cf91dac07" (UID: "ee7cc97d-228d-4f75-a3b9-914cf91dac07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:09.091752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.091721    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.091752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.091749    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.091752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.091759    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztcr\" (UniqueName: \"kubernetes.io/projected/ee7cc97d-228d-4f75-a3b9-914cf91dac07-kube-api-access-bztcr\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.092016 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.091770    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7cc97d-228d-4f75-a3b9-914cf91dac07-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.096977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.096938    6695 generic.go:356] "Generic (PLEG): container finished" podID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerID="2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d" exitCode=0
Oct 30 22:11:09.097164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.097020    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"
Oct 30 22:11:09.097164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.097060    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" event={"ID":"ee7cc97d-228d-4f75-a3b9-914cf91dac07","Type":"ContainerDied","Data":"2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d"}
Oct 30 22:11:09.097295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.097182    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67" event={"ID":"ee7cc97d-228d-4f75-a3b9-914cf91dac07","Type":"ContainerDied","Data":"ea2b5513475e188a4477deab518637e138322163833bfa66113945a23be444f1"}
Oct 30 22:11:09.097295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.097222    6695 scope.go:122] "RemoveContainer" containerID="2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d"
Oct 30 22:11:09.112246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.112222    6695 scope.go:122] "RemoveContainer" containerID="2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d"
Oct 30 22:11:09.112587 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:09.112562    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d\": container with ID starting with 2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d not found: ID does not exist" containerID="2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d"
Oct 30 22:11:09.112734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.112638    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d"} err="failed to get container status \"2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d\": rpc error: code = NotFound desc = could not find container \"2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d\": container with ID starting with 2707169b49be3e8172e91933c2ebc4f448c4385d411fe395c8a65009c37c1b1d not found: ID does not exist"
Oct 30 22:11:09.158064 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.157994    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"]
Oct 30 22:11:09.163411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.163388    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4cp67"]
Oct 30 22:11:09.222926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:09.222890    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" path="/var/lib/kubelet/pods/ee7cc97d-228d-4f75-a3b9-914cf91dac07/volumes"
Oct 30 22:11:12.676646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:12.676593    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:12.677072 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:12.676620    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:19.830198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:19.830161    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"]
Oct 30 22:11:19.830636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:19.830365    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy" containerID="cri-o://ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed" gracePeriod=30
Oct 30 22:11:22.676638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:22.676570    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:22.677250 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:22.676597    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:25.122297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.122271    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:11:25.137262 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137230    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137268    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137332    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137365    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137457    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137498    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137770 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137749    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137840 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137783    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.137911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137886    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:25.137973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137911    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:25.137973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.137894    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token\") pod \"806f2d5a-e7ee-48a8-b90d-77b761d67260\" (UID: \"806f2d5a-e7ee-48a8-b90d-77b761d67260\") "
Oct 30 22:11:25.138092 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138026    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:25.138092 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138063    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:25.138236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138215    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:25.138557 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138540    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.138686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138560    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806f2d5a-e7ee-48a8-b90d-77b761d67260-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.138686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138569    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.138686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138581    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.138686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.138591    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.139756 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.139732    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:25.140135 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.140118    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:25.140217 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.140156    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "kube-api-access-dbm4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:25.140282 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.140237    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token" pod "806f2d5a-e7ee-48a8-b90d-77b761d67260" (UID: "806f2d5a-e7ee-48a8-b90d-77b761d67260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:25.231418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.231371    6695 generic.go:356] "Generic (PLEG): container finished" podID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerID="ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed" exitCode=0
Oct 30 22:11:25.231580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.231425    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"
Oct 30 22:11:25.231580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.231436    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" event={"ID":"806f2d5a-e7ee-48a8-b90d-77b761d67260","Type":"ContainerDied","Data":"ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed"}
Oct 30 22:11:25.231580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.231496    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64" event={"ID":"806f2d5a-e7ee-48a8-b90d-77b761d67260","Type":"ContainerDied","Data":"90132d21a842dc8068406c467fcb0fa7b37701b4142fa997c951e4ad1303652e"}
Oct 30 22:11:25.231580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.231529    6695 scope.go:122] "RemoveContainer" containerID="ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed"
Oct 30 22:11:25.246396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.246361    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.246572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.246403    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.246572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.246417    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806f2d5a-e7ee-48a8-b90d-77b761d67260-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.246572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.246432    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbm4t\" (UniqueName: \"kubernetes.io/projected/806f2d5a-e7ee-48a8-b90d-77b761d67260-kube-api-access-dbm4t\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:25.268334 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.268309    6695 scope.go:122] "RemoveContainer" containerID="ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed"
Oct 30 22:11:25.268707 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:25.268676    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed\": container with ID starting with ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed not found: ID does not exist" containerID="ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed"
Oct 30 22:11:25.268798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.268719    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed"} err="failed to get container status \"ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed\": rpc error: code = NotFound desc = could not find container \"ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed\": container with ID starting with ecb3613b17a96dd5e59d596f848386e52330ebb3151feced2a774048a4edb0ed not found: ID does not exist"
Oct 30 22:11:25.297854 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.297815    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"]
Oct 30 22:11:25.302288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:25.302258    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-jjx64"]
Oct 30 22:11:27.216303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.216266    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" path="/var/lib/kubelet/pods/806f2d5a-e7ee-48a8-b90d-77b761d67260/volumes"
Oct 30 22:11:27.691079 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.691044    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:11:27.725918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.725891    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:11:27.787344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.787316    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume\") pod \"cd793a5f-8112-4109-9c13-3f393b019c13\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") "
Oct 30 22:11:27.787547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.787368    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl\" (UniqueName: \"kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl\") pod \"cd793a5f-8112-4109-9c13-3f393b019c13\" (UID: \"cd793a5f-8112-4109-9c13-3f393b019c13\") "
Oct 30 22:11:27.787663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.787567    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd\" (UniqueName: \"kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd\") pod \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") "
Oct 30 22:11:27.787746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.787697    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume\") pod \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\" (UID: \"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30\") "
Oct 30 22:11:27.788262 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.788225    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume" pod "81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" (UID: "81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:27.788384 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.788363    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume" pod "cd793a5f-8112-4109-9c13-3f393b019c13" (UID: "cd793a5f-8112-4109-9c13-3f393b019c13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:27.790239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.790212    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd" pod "81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" (UID: "81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30"). InnerVolumeSpecName "kube-api-access-v44vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:27.790360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.790307    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl" pod "cd793a5f-8112-4109-9c13-3f393b019c13" (UID: "cd793a5f-8112-4109-9c13-3f393b019c13"). InnerVolumeSpecName "kube-api-access-xftwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:27.890221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.890188    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44vd\" (UniqueName: \"kubernetes.io/projected/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-kube-api-access-v44vd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:27.890221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.890218    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:27.890221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.890229    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd793a5f-8112-4109-9c13-3f393b019c13-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:27.890488 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:27.890237    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftwl\" (UniqueName: \"kubernetes.io/projected/cd793a5f-8112-4109-9c13-3f393b019c13-kube-api-access-xftwl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:28.258521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.258432    6695 generic.go:356] "Generic (PLEG): container finished" podID="cd793a5f-8112-4109-9c13-3f393b019c13" containerID="3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489" exitCode=0
Oct 30 22:11:28.258521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.258499    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"
Oct 30 22:11:28.259149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.258499    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" event={"ID":"cd793a5f-8112-4109-9c13-3f393b019c13","Type":"ContainerDied","Data":"3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489"}
Oct 30 22:11:28.259149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.258641    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c" event={"ID":"cd793a5f-8112-4109-9c13-3f393b019c13","Type":"ContainerDied","Data":"3766f2e30c783d09307bd371456493f7f77de8c58cc2ad3b1a617b5622781171"}
Oct 30 22:11:28.259149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.258674    6695 scope.go:122] "RemoveContainer" containerID="3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489"
Oct 30 22:11:28.262387 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.262365    6695 generic.go:356] "Generic (PLEG): container finished" podID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerID="2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5" exitCode=0
Oct 30 22:11:28.262525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.262408    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" event={"ID":"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30","Type":"ContainerDied","Data":"2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5"}
Oct 30 22:11:28.262525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.262425    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"
Oct 30 22:11:28.262525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.262449    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956" event={"ID":"81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30","Type":"ContainerDied","Data":"e345f28d893db96d7483c8974a3b13b4524d20e9bc242d36a282d9dda1820b65"}
Oct 30 22:11:28.272975 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.272954    6695 scope.go:122] "RemoveContainer" containerID="3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489"
Oct 30 22:11:28.273251 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:28.273229    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489\": container with ID starting with 3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489 not found: ID does not exist" containerID="3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489"
Oct 30 22:11:28.273303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.273259    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489"} err="failed to get container status \"3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489\": rpc error: code = NotFound desc = could not find container \"3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489\": container with ID starting with 3cb4e43e05af81612f8d974617a1727b45c8743f19889702c5ca8c272df8c489 not found: ID does not exist"
Oct 30 22:11:28.273303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.273276    6695 scope.go:122] "RemoveContainer" containerID="2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5"
Oct 30 22:11:28.284095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.284071    6695 scope.go:122] "RemoveContainer" containerID="2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5"
Oct 30 22:11:28.284394 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:28.284372    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5\": container with ID starting with 2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5 not found: ID does not exist" containerID="2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5"
Oct 30 22:11:28.284444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.284402    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5"} err="failed to get container status \"2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5\": rpc error: code = NotFound desc = could not find container \"2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5\": container with ID starting with 2b2678c90a041aa0ffbcb7392bc1b95daa1db08c7cbaddcffae28a5862bf3eb5 not found: ID does not exist"
Oct 30 22:11:28.332039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.332007    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"]
Oct 30 22:11:28.344485 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.344455    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8d53e0-55cd7c9cdc-xc956"]
Oct 30 22:11:28.401551 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.401510    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"]
Oct 30 22:11:28.409254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:28.409225    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8d53e0-d6f7449ff-pxp8c"]
Oct 30 22:11:29.215866 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:29.215829    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" path="/var/lib/kubelet/pods/81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30/volumes"
Oct 30 22:11:29.216291 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:29.216277    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" path="/var/lib/kubelet/pods/cd793a5f-8112-4109-9c13-3f393b019c13/volumes"
Oct 30 22:11:29.278571 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:29.278538    6695 scope.go:122] "RemoveContainer" containerID="5ff115e20404b3183bdd23c5e5f77295a741f0949335d4f034eb3178cbf92ce6"
Oct 30 22:11:29.288927 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:29.288905    6695 scope.go:122] "RemoveContainer" containerID="ec967ca957f5c9b0f1e5b013a78fd3f921d9617a5de282e8056425bc73a16453"
Oct 30 22:11:29.299073 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:29.299051    6695 scope.go:122] "RemoveContainer" containerID="d2faaf1df73223a895aa1d35d3a710c409b8c7b258f7d0c5699512e8e5a70828"
Oct 30 22:11:31.074467 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:11:31.074425    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:11:35.038876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:35.038837    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"]
Oct 30 22:11:35.053546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:35.050312    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8d53e0-dkldg"]
Oct 30 22:11:35.216653 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:35.216617    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08926c1-5562-49b8-8ad7-c297592a805b" path="/var/lib/kubelet/pods/b08926c1-5562-49b8-8ad7-c297592a805b/volumes"
Oct 30 22:11:42.240645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.233854    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"]
Oct 30 22:11:42.240645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.234262    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy" containerID="cri-o://03da5571efda7cfa88311b6d442a2ad18427e8a48775230dc5652c545b9fb0d6" gracePeriod=30
Oct 30 22:11:42.804904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.804861    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"]
Oct 30 22:11:42.805632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805599    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b08926c1-5562-49b8-8ad7-c297592a805b" containerName="revoke-keys"
Oct 30 22:11:42.805632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805630    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08926c1-5562-49b8-8ad7-c297592a805b" containerName="revoke-keys"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805644    6695 cpu_manager.go:414] "containerMap: removing container" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" containerName="payload-pre-processing"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805649    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" containerName="payload-pre-processing"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805667    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805673    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805680    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805684    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805707    6695 cpu_manager.go:414] "containerMap: removing container" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerName="payload-processing"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805712    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerName="payload-processing"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805723    6695 cpu_manager.go:414] "containerMap: removing container" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy"
Oct 30 22:11:42.805728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805728    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805737    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0f0c618f-5bd8-4144-a480-f1c40234ce00" containerName="main"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805741    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0c618f-5bd8-4144-a480-f1c40234ce00" containerName="main"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805748    6695 cpu_manager.go:414] "containerMap: removing container" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805753    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805869    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08926c1-5562-49b8-8ad7-c297592a805b" containerName="revoke-keys"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805877    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd793a5f-8112-4109-9c13-3f393b019c13" containerName="payload-pre-processing"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805886    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd5c64d-736c-4fff-9f58-e0f2fdfbdcbe" containerName="istio-proxy"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805894    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5675ebbf-042a-4a91-ab8c-1adacaf45777" containerName="istio-proxy"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805903    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e3ccb3-522c-46c3-9b3a-a4d68d9bdb30" containerName="payload-processing"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805911    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7cc97d-228d-4f75-a3b9-914cf91dac07" containerName="istio-proxy"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805917    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0c618f-5bd8-4144-a480-f1c40234ce00" containerName="main"
Oct 30 22:11:42.806063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.805922    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="806f2d5a-e7ee-48a8-b90d-77b761d67260" containerName="istio-proxy"
Oct 30 22:11:42.811591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.811570    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.825978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.825945    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"]
Oct 30 22:11:42.828669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.828643    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:11:42.828836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.828756    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:42.829297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.829278    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:42.850837 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.850801    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:42.866625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.865747    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcjt\" (UniqueName: \"kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.866625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.865879    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.866625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.866010    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.967420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.967375    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcjt\" (UniqueName: \"kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.967665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.967540    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.967761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.967689    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.968398 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.968375    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.970007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.969983    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:42.986013 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:42.985984    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcjt\" (UniqueName: \"kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt\") pod \"maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:43.127413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:43.127326    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:43.436232 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:43.436200    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"]
Oct 30 22:11:43.438839 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:11:43.438810    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd520f944_8e01_4603_8db2_5013ee582366.slice/crio-20b78614c7cf07d4a6b079b06db3c1e4c5d8101c057e12f44df621d3eef22dc4 WatchSource:0}: Error finding container 20b78614c7cf07d4a6b079b06db3c1e4c5d8101c057e12f44df621d3eef22dc4: Status 404 returned error can't find the container with id 20b78614c7cf07d4a6b079b06db3c1e4c5d8101c057e12f44df621d3eef22dc4
Oct 30 22:11:44.426519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:44.426483    6695 generic.go:356] "Generic (PLEG): container finished" podID="d520f944-8e01-4603-8db2-5013ee582366" containerID="482d13fd5e3f1dfcb4383e34d9daba9fe67d34d85e2e35bb7288b744e6346f3e" exitCode=0
Oct 30 22:11:44.426746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:44.426572    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf" event={"ID":"d520f944-8e01-4603-8db2-5013ee582366","Type":"ContainerDied","Data":"482d13fd5e3f1dfcb4383e34d9daba9fe67d34d85e2e35bb7288b744e6346f3e"}
Oct 30 22:11:44.426746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:44.426645    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf" event={"ID":"d520f944-8e01-4603-8db2-5013ee582366","Type":"ContainerStarted","Data":"20b78614c7cf07d4a6b079b06db3c1e4c5d8101c057e12f44df621d3eef22dc4"}
Oct 30 22:11:45.627436 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.627408    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:45.720010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.719931    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt\" (UniqueName: \"kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt\") pod \"d520f944-8e01-4603-8db2-5013ee582366\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") "
Oct 30 22:11:45.720272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.720245    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca\") pod \"d520f944-8e01-4603-8db2-5013ee582366\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") "
Oct 30 22:11:45.720422 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.720293    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca\") pod \"d520f944-8e01-4603-8db2-5013ee582366\" (UID: \"d520f944-8e01-4603-8db2-5013ee582366\") "
Oct 30 22:11:45.720646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.720620    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca" pod "d520f944-8e01-4603-8db2-5013ee582366" (UID: "d520f944-8e01-4603-8db2-5013ee582366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:45.720946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.720926    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520f944-8e01-4603-8db2-5013ee582366-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:45.722289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.722269    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt" pod "d520f944-8e01-4603-8db2-5013ee582366" (UID: "d520f944-8e01-4603-8db2-5013ee582366"). InnerVolumeSpecName "kube-api-access-nqcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:45.722366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.722311    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca" pod "d520f944-8e01-4603-8db2-5013ee582366" (UID: "d520f944-8e01-4603-8db2-5013ee582366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:45.822621 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.822575    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520f944-8e01-4603-8db2-5013ee582366-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:45.822823 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:45.822638    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcjt\" (UniqueName: \"kubernetes.io/projected/d520f944-8e01-4603-8db2-5013ee582366-kube-api-access-nqcjt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:46.444896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:46.444850    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"
Oct 30 22:11:46.444896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:46.444878    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf" event={"ID":"d520f944-8e01-4603-8db2-5013ee582366","Type":"ContainerDied","Data":"20b78614c7cf07d4a6b079b06db3c1e4c5d8101c057e12f44df621d3eef22dc4"}
Oct 30 22:11:46.445125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:46.444927    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b78614c7cf07d4a6b079b06db3c1e4c5d8101c057e12f44df621d3eef22dc4"
Oct 30 22:11:47.457279 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.457233    6695 generic.go:356] "Generic (PLEG): container finished" podID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerID="03da5571efda7cfa88311b6d442a2ad18427e8a48775230dc5652c545b9fb0d6" exitCode=0
Oct 30 22:11:47.457784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.457308    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" event={"ID":"49803228-087d-4b4c-8d79-fcc236abe8f8","Type":"ContainerDied","Data":"03da5571efda7cfa88311b6d442a2ad18427e8a48775230dc5652c545b9fb0d6"}
Oct 30 22:11:47.457784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.457393    6695 scope.go:122] "RemoveContainer" containerID="1be1b9ec6a125ff27eabbc28741dfe24fa85b33b13a8de6a4f7a02e5c69c7c6a"
Oct 30 22:11:47.522940 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.522915    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:11:47.650737 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.650699    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.650928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.650836    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.650928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.650908    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.650931    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.650960    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.650993    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651048    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651100    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651149    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d\") pod \"49803228-087d-4b4c-8d79-fcc236abe8f8\" (UID: \"49803228-087d-4b4c-8d79-fcc236abe8f8\") "
Oct 30 22:11:47.651430 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651316    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:47.651430 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651331    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:47.651430 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651338    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:47.651640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651553    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:47.651640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651569    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:47.651901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651879    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.651978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651909    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.651978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651926    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.651978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651942    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.651978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.651956    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49803228-087d-4b4c-8d79-fcc236abe8f8-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.653103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.653074    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:47.653563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.653540    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:47.653704 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.653683    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:47.653762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.653744    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d" pod "49803228-087d-4b4c-8d79-fcc236abe8f8" (UID: "49803228-087d-4b4c-8d79-fcc236abe8f8"). InnerVolumeSpecName "kube-api-access-bvt9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:47.753405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.753374    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.753405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.753402    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvt9d\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-kube-api-access-bvt9d\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.753405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.753413    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:47.753687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:47.753422    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49803228-087d-4b4c-8d79-fcc236abe8f8-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:48.466955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:48.466926    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"
Oct 30 22:11:48.467423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:48.466974    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt" event={"ID":"49803228-087d-4b4c-8d79-fcc236abe8f8","Type":"ContainerDied","Data":"f95b1dccc2811681dd39009de9b137fae83d8aa4a94cc91bd86387391a7cbe1a"}
Oct 30 22:11:48.467423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:48.467031    6695 scope.go:122] "RemoveContainer" containerID="03da5571efda7cfa88311b6d442a2ad18427e8a48775230dc5652c545b9fb0d6"
Oct 30 22:11:48.530459 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:48.530431    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"]
Oct 30 22:11:48.541660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:48.541634    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d53e0-openshift-default-7f6f66b869-tl7kt"]
Oct 30 22:11:49.220868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.220828    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" path="/var/lib/kubelet/pods/49803228-087d-4b4c-8d79-fcc236abe8f8/volumes"
Oct 30 22:11:49.343764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.343726    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"]
Oct 30 22:11:49.344441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344428    6695 cpu_manager.go:414] "containerMap: removing container" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy"
Oct 30 22:11:49.344493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344443    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy"
Oct 30 22:11:49.344493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344489    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d520f944-8e01-4603-8db2-5013ee582366" containerName="revoke-keys"
Oct 30 22:11:49.344569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344496    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d520f944-8e01-4603-8db2-5013ee582366" containerName="revoke-keys"
Oct 30 22:11:49.344631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344619    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d520f944-8e01-4603-8db2-5013ee582366" containerName="revoke-keys"
Oct 30 22:11:49.344677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344638    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy"
Oct 30 22:11:49.344677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.344646    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy"
Oct 30 22:11:49.349660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.349638    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.364747 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.364721    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:49.364919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.364826    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:49.365918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.365890    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"]
Oct 30 22:11:49.372424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.372394    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:49.372556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.372478    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:11:49.476945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.476840    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqsq\" (UniqueName: \"kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.477480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.477034    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.477480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.477103    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.578775 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.578721    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.578971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.578919    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqsq\" (UniqueName: \"kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.579080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.579061    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.579892 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.579858    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.581241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.581208    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.598003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.597971    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqsq\" (UniqueName: \"kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq\") pod \"maas-api-revoke-sub-7e17c031-g5d9p\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.664465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.664426    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:49.972432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:49.972405    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"]
Oct 30 22:11:49.975005 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:11:49.974966    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d91286_bcb2_465a_b179_1a6c5c8c4d30.slice/crio-1fcc148d3f21702cb71840b5fb06049eee1ab33463abded97f34eb4cb2efb4fe WatchSource:0}: Error finding container 1fcc148d3f21702cb71840b5fb06049eee1ab33463abded97f34eb4cb2efb4fe: Status 404 returned error can't find the container with id 1fcc148d3f21702cb71840b5fb06049eee1ab33463abded97f34eb4cb2efb4fe
Oct 30 22:11:50.491352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:50.491301    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p" event={"ID":"65d91286-bcb2-465a-b179-1a6c5c8c4d30","Type":"ContainerStarted","Data":"d16bc9b8f41b73bf02590c8c47344bb34cc8c1c5fe6876b97a4db6250cd75323"}
Oct 30 22:11:50.492224 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:50.492196    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p" event={"ID":"65d91286-bcb2-465a-b179-1a6c5c8c4d30","Type":"ContainerStarted","Data":"1fcc148d3f21702cb71840b5fb06049eee1ab33463abded97f34eb4cb2efb4fe"}
Oct 30 22:11:50.531456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:50.531396    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p" podStartSLOduration=1.531382297 podStartE2EDuration="1.531382297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:50.530393146 +0000 UTC m=+3693.452851444" watchObservedRunningTime="2026-10-30 22:11:50.531382297 +0000 UTC m=+3693.453840588"
Oct 30 22:11:51.501539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:51.501496    6695 generic.go:356] "Generic (PLEG): container finished" podID="65d91286-bcb2-465a-b179-1a6c5c8c4d30" containerID="d16bc9b8f41b73bf02590c8c47344bb34cc8c1c5fe6876b97a4db6250cd75323" exitCode=0
Oct 30 22:11:51.502008 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:51.501558    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p" event={"ID":"65d91286-bcb2-465a-b179-1a6c5c8c4d30","Type":"ContainerDied","Data":"d16bc9b8f41b73bf02590c8c47344bb34cc8c1c5fe6876b97a4db6250cd75323"}
Oct 30 22:11:52.669535 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.669507    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:52.724617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.724567    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca\") pod \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") "
Oct 30 22:11:52.724811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.724763    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq\" (UniqueName: \"kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq\") pod \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") "
Oct 30 22:11:52.724891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.724875    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca\") pod \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\" (UID: \"65d91286-bcb2-465a-b179-1a6c5c8c4d30\") "
Oct 30 22:11:52.725299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.725269    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca" pod "65d91286-bcb2-465a-b179-1a6c5c8c4d30" (UID: "65d91286-bcb2-465a-b179-1a6c5c8c4d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:52.725718 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.725695    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:52.726915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.726888    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq" pod "65d91286-bcb2-465a-b179-1a6c5c8c4d30" (UID: "65d91286-bcb2-465a-b179-1a6c5c8c4d30"). InnerVolumeSpecName "kube-api-access-whqsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:52.726990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.726968    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca" pod "65d91286-bcb2-465a-b179-1a6c5c8c4d30" (UID: "65d91286-bcb2-465a-b179-1a6c5c8c4d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:52.827741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.827697    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d91286-bcb2-465a-b179-1a6c5c8c4d30-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:52.827741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:52.827741    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqsq\" (UniqueName: \"kubernetes.io/projected/65d91286-bcb2-465a-b179-1a6c5c8c4d30-kube-api-access-whqsq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:53.287451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:53.287407    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"]
Oct 30 22:11:53.339493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:53.339429    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"]
Oct 30 22:11:53.517935 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:53.517835    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"
Oct 30 22:11:53.517935 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:53.517866    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p" event={"ID":"65d91286-bcb2-465a-b179-1a6c5c8c4d30","Type":"ContainerDied","Data":"1fcc148d3f21702cb71840b5fb06049eee1ab33463abded97f34eb4cb2efb4fe"}
Oct 30 22:11:53.517935 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:53.517925    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fcc148d3f21702cb71840b5fb06049eee1ab33463abded97f34eb4cb2efb4fe"
Oct 30 22:11:58.288321 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.288275    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s" podUID="187b23fa-08f8-4d55-8fe6-4ec57bc52824" containerName="payload-processing" containerID="cri-o://48dbaf4fe7f2f9e294a7304c2443d225ec2cfca93c544b89f57eb3cb2eec4176" gracePeriod=25
Oct 30 22:11:58.340114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.340075    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerName="payload-pre-processing" containerID="cri-o://645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25" gracePeriod=25
Oct 30 22:11:58.559427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.559391    6695 generic.go:356] "Generic (PLEG): container finished" podID="187b23fa-08f8-4d55-8fe6-4ec57bc52824" containerID="48dbaf4fe7f2f9e294a7304c2443d225ec2cfca93c544b89f57eb3cb2eec4176" exitCode=0
Oct 30 22:11:58.559632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.559474    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s" event={"ID":"187b23fa-08f8-4d55-8fe6-4ec57bc52824","Type":"ContainerDied","Data":"48dbaf4fe7f2f9e294a7304c2443d225ec2cfca93c544b89f57eb3cb2eec4176"}
Oct 30 22:11:58.559632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.559544    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s" event={"ID":"187b23fa-08f8-4d55-8fe6-4ec57bc52824","Type":"ContainerDied","Data":"460afa73beea1fb2a4c8a3267f857bc54838949a3801b8a4f9104f7526f2dd1c"}
Oct 30 22:11:58.559632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.559567    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460afa73beea1fb2a4c8a3267f857bc54838949a3801b8a4f9104f7526f2dd1c"
Oct 30 22:11:58.565205 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.565184    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:11:58.612286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.612256    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9\" (UniqueName: \"kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9\") pod \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") "
Oct 30 22:11:58.612465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.612444    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume\") pod \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\" (UID: \"187b23fa-08f8-4d55-8fe6-4ec57bc52824\") "
Oct 30 22:11:58.612810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.612784    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume" pod "187b23fa-08f8-4d55-8fe6-4ec57bc52824" (UID: "187b23fa-08f8-4d55-8fe6-4ec57bc52824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:58.613057 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.613028    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b23fa-08f8-4d55-8fe6-4ec57bc52824-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:58.614489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.614462    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9" pod "187b23fa-08f8-4d55-8fe6-4ec57bc52824" (UID: "187b23fa-08f8-4d55-8fe6-4ec57bc52824"). InnerVolumeSpecName "kube-api-access-qktb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:58.714283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:58.714247    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qktb9\" (UniqueName: \"kubernetes.io/projected/187b23fa-08f8-4d55-8fe6-4ec57bc52824-kube-api-access-qktb9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:11:59.568725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:59.568692    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"
Oct 30 22:11:59.621089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:59.621054    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"]
Oct 30 22:11:59.632703 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:11:59.632673    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6537e6-7c58ff7947-krp7s"]
Oct 30 22:12:00.814727 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.814689    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"]
Oct 30 22:12:00.815666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815643    6695 cpu_manager.go:414] "containerMap: removing container" podUID="187b23fa-08f8-4d55-8fe6-4ec57bc52824" containerName="payload-processing"
Oct 30 22:12:00.815666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815667    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187b23fa-08f8-4d55-8fe6-4ec57bc52824" containerName="payload-processing"
Oct 30 22:12:00.815863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815691    6695 cpu_manager.go:414] "containerMap: removing container" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy"
Oct 30 22:12:00.815863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815697    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49803228-087d-4b4c-8d79-fcc236abe8f8" containerName="istio-proxy"
Oct 30 22:12:00.815863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815723    6695 cpu_manager.go:414] "containerMap: removing container" podUID="65d91286-bcb2-465a-b179-1a6c5c8c4d30" containerName="revoke-keys"
Oct 30 22:12:00.815863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815732    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d91286-bcb2-465a-b179-1a6c5c8c4d30" containerName="revoke-keys"
Oct 30 22:12:00.816095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815937    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="187b23fa-08f8-4d55-8fe6-4ec57bc52824" containerName="payload-processing"
Oct 30 22:12:00.816095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.815959    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d91286-bcb2-465a-b179-1a6c5c8c4d30" containerName="revoke-keys"
Oct 30 22:12:00.819900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.819877    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:00.842278 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.842239    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"]
Oct 30 22:12:00.846075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.846038    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:00.846075 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.846068    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:00.846286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.846068    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:00.846286 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.846156    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:12:00.955307 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.955271    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:00.955510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.955404    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mws9\" (UniqueName: \"kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:00.955672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:00.955648    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.057556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.057517    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.057826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.057632    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.057826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.057723    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mws9\" (UniqueName: \"kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.058276 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.058248    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.060152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.060123    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.078175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.078102    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mws9\" (UniqueName: \"kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9\") pod \"maas-api-revoke-sub-7e031d2f-z7sbl\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.081126 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:01.081094    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:12:01.134588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.134554    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:01.237229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.237195    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187b23fa-08f8-4d55-8fe6-4ec57bc52824" path="/var/lib/kubelet/pods/187b23fa-08f8-4d55-8fe6-4ec57bc52824/volumes"
Oct 30 22:12:01.449665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.449636    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"]
Oct 30 22:12:01.452375 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:01.452333    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f1e7a3_a725_44b8_a54d_51bb006ab201.slice/crio-840150336db1d90f96f3801b55ed99686b32fcc67fe2dccea7cb2a12d41bc4fe WatchSource:0}: Error finding container 840150336db1d90f96f3801b55ed99686b32fcc67fe2dccea7cb2a12d41bc4fe: Status 404 returned error can't find the container with id 840150336db1d90f96f3801b55ed99686b32fcc67fe2dccea7cb2a12d41bc4fe
Oct 30 22:12:01.588841 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:01.588800    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl" event={"ID":"29f1e7a3-a725-44b8-a54d-51bb006ab201","Type":"ContainerStarted","Data":"840150336db1d90f96f3801b55ed99686b32fcc67fe2dccea7cb2a12d41bc4fe"}
Oct 30 22:12:02.375741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.375692    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"]
Oct 30 22:12:02.380561 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.380534    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.397596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.397565    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"]
Oct 30 22:12:02.481236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.481200    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.481440 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.481413    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.481572 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.481553    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpl6\" (UniqueName: \"kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.582491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.582392    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.582491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.582485    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpl6\" (UniqueName: \"kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.582791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.582533    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.583030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.583005    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.584899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.584874    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.600177 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.600143    6695 generic.go:356] "Generic (PLEG): container finished" podID="29f1e7a3-a725-44b8-a54d-51bb006ab201" containerID="b4edb4e9000f33de408760a75d430e9e7500da919a6d09e5a18532b3fc606a61" exitCode=0
Oct 30 22:12:02.600385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.600236    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl" event={"ID":"29f1e7a3-a725-44b8-a54d-51bb006ab201","Type":"ContainerDied","Data":"b4edb4e9000f33de408760a75d430e9e7500da919a6d09e5a18532b3fc606a61"}
Oct 30 22:12:02.602668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.602646    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpl6\" (UniqueName: \"kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6\") pod \"maas-api-revoke-sub-2dc05653-cqf2m\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:02.709349 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:02.709310    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:03.023275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.023158    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"]
Oct 30 22:12:03.026341 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:03.026305    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12cca9f_b84d_4019_9f79_8866f7ac5c1c.slice/crio-44110471f2fffff8304be992b8f717f89562e26c149229f82d449c71132c0ace WatchSource:0}: Error finding container 44110471f2fffff8304be992b8f717f89562e26c149229f82d449c71132c0ace: Status 404 returned error can't find the container with id 44110471f2fffff8304be992b8f717f89562e26c149229f82d449c71132c0ace
Oct 30 22:12:03.610817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.610710    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m" event={"ID":"c12cca9f-b84d-4019-9f79-8866f7ac5c1c","Type":"ContainerStarted","Data":"f1a9e27bf26ac9b875a85d44e70e17b1d11556415ffe46304dea0559edce8acd"}
Oct 30 22:12:03.610817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.610785    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m" event={"ID":"c12cca9f-b84d-4019-9f79-8866f7ac5c1c","Type":"ContainerStarted","Data":"44110471f2fffff8304be992b8f717f89562e26c149229f82d449c71132c0ace"}
Oct 30 22:12:03.650398 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.650319    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m" podStartSLOduration=1.6503015749999999 podStartE2EDuration="1.650301575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:03.648678368 +0000 UTC m=+3706.571136663" watchObservedRunningTime="2026-10-30 22:12:03.650301575 +0000 UTC m=+3706.572759872"
Oct 30 22:12:03.767004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.766975    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:03.899879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.899789    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9\" (UniqueName: \"kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9\") pod \"29f1e7a3-a725-44b8-a54d-51bb006ab201\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") "
Oct 30 22:12:03.900066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.899981    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca\") pod \"29f1e7a3-a725-44b8-a54d-51bb006ab201\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") "
Oct 30 22:12:03.900142 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.900128    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca\") pod \"29f1e7a3-a725-44b8-a54d-51bb006ab201\" (UID: \"29f1e7a3-a725-44b8-a54d-51bb006ab201\") "
Oct 30 22:12:03.900496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.900471    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca" pod "29f1e7a3-a725-44b8-a54d-51bb006ab201" (UID: "29f1e7a3-a725-44b8-a54d-51bb006ab201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:03.900735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.900716    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:03.902070 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.902049    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca" pod "29f1e7a3-a725-44b8-a54d-51bb006ab201" (UID: "29f1e7a3-a725-44b8-a54d-51bb006ab201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:03.902230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:03.902215    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9" pod "29f1e7a3-a725-44b8-a54d-51bb006ab201" (UID: "29f1e7a3-a725-44b8-a54d-51bb006ab201"). InnerVolumeSpecName "kube-api-access-7mws9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:04.001952 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.001913    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29f1e7a3-a725-44b8-a54d-51bb006ab201-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:04.001952 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.001950    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mws9\" (UniqueName: \"kubernetes.io/projected/29f1e7a3-a725-44b8-a54d-51bb006ab201-kube-api-access-7mws9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:04.624740 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.624705    6695 generic.go:356] "Generic (PLEG): container finished" podID="c12cca9f-b84d-4019-9f79-8866f7ac5c1c" containerID="f1a9e27bf26ac9b875a85d44e70e17b1d11556415ffe46304dea0559edce8acd" exitCode=0
Oct 30 22:12:04.625293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.624780    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m" event={"ID":"c12cca9f-b84d-4019-9f79-8866f7ac5c1c","Type":"ContainerDied","Data":"f1a9e27bf26ac9b875a85d44e70e17b1d11556415ffe46304dea0559edce8acd"}
Oct 30 22:12:04.626501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.626474    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl" event={"ID":"29f1e7a3-a725-44b8-a54d-51bb006ab201","Type":"ContainerDied","Data":"840150336db1d90f96f3801b55ed99686b32fcc67fe2dccea7cb2a12d41bc4fe"}
Oct 30 22:12:04.626689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.626507    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e031d2f-z7sbl"
Oct 30 22:12:04.626689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:04.626521    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="840150336db1d90f96f3801b55ed99686b32fcc67fe2dccea7cb2a12d41bc4fe"
Oct 30 22:12:05.822346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.822318    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:05.929146 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.929043    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca\") pod \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") "
Oct 30 22:12:05.929382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.929255    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6\" (UniqueName: \"kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6\") pod \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") "
Oct 30 22:12:05.929382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.929374    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca\") pod \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\" (UID: \"c12cca9f-b84d-4019-9f79-8866f7ac5c1c\") "
Oct 30 22:12:05.930199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.930169    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca" pod "c12cca9f-b84d-4019-9f79-8866f7ac5c1c" (UID: "c12cca9f-b84d-4019-9f79-8866f7ac5c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:05.935230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.935200    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6" pod "c12cca9f-b84d-4019-9f79-8866f7ac5c1c" (UID: "c12cca9f-b84d-4019-9f79-8866f7ac5c1c"). InnerVolumeSpecName "kube-api-access-2xpl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:05.935538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:05.935506    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca" pod "c12cca9f-b84d-4019-9f79-8866f7ac5c1c" (UID: "c12cca9f-b84d-4019-9f79-8866f7ac5c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:06.030920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.030877    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xpl6\" (UniqueName: \"kubernetes.io/projected/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-kube-api-access-2xpl6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:06.030920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.030913    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:06.030920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.030924    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12cca9f-b84d-4019-9f79-8866f7ac5c1c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:06.084912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.084869    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:06.654968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.654934    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m"
Oct 30 22:12:06.655188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.654937    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dc05653-cqf2m" event={"ID":"c12cca9f-b84d-4019-9f79-8866f7ac5c1c","Type":"ContainerDied","Data":"44110471f2fffff8304be992b8f717f89562e26c149229f82d449c71132c0ace"}
Oct 30 22:12:06.655188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:06.655068    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44110471f2fffff8304be992b8f717f89562e26c149229f82d449c71132c0ace"
Oct 30 22:12:16.105669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:16.105620    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:16.355037 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:16.355001    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"]
Oct 30 22:12:16.366520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:16.366425    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6537e6-vb4cf"]
Oct 30 22:12:17.066910 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.066867    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"]
Oct 30 22:12:17.067637 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.067621    6695 cpu_manager.go:414] "containerMap: removing container" podUID="29f1e7a3-a725-44b8-a54d-51bb006ab201" containerName="revoke-keys"
Oct 30 22:12:17.067693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.067638    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f1e7a3-a725-44b8-a54d-51bb006ab201" containerName="revoke-keys"
Oct 30 22:12:17.067693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.067655    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c12cca9f-b84d-4019-9f79-8866f7ac5c1c" containerName="revoke-keys"
Oct 30 22:12:17.067693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.067660    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12cca9f-b84d-4019-9f79-8866f7ac5c1c" containerName="revoke-keys"
Oct 30 22:12:17.067800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.067764    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f1e7a3-a725-44b8-a54d-51bb006ab201" containerName="revoke-keys"
Oct 30 22:12:17.067800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.067772    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12cca9f-b84d-4019-9f79-8866f7ac5c1c" containerName="revoke-keys"
Oct 30 22:12:17.071860 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.071837    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.092959 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.092928    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"]
Oct 30 22:12:17.099830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.099796    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:17.100007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.099893    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:12:17.110979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.110939    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:17.110979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.110949    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:17.139459 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.139411    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkdgs\" (UniqueName: \"kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.139795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.139518    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.139795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.139582    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.185124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.185093    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"]
Oct 30 22:12:17.216175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.216142    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d520f944-8e01-4603-8db2-5013ee582366" path="/var/lib/kubelet/pods/d520f944-8e01-4603-8db2-5013ee582366/volumes"
Oct 30 22:12:17.241614 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.241534    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkdgs\" (UniqueName: \"kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.241840 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.241679    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.241840 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.241746    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.242515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.242494    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.244407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.244365    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.316559 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.316510    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkdgs\" (UniqueName: \"kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs\") pod \"maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.386156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.386074    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:17.697994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.697967    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"]
Oct 30 22:12:17.700639 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:17.700593    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94aaca84_9a1a_45f3_921d_b48c4675bbec.slice/crio-e3d4d5ea2125253bf639deba9e32bb22f2abc799378c0d70fb98fd8108f54469 WatchSource:0}: Error finding container e3d4d5ea2125253bf639deba9e32bb22f2abc799378c0d70fb98fd8108f54469: Status 404 returned error can't find the container with id e3d4d5ea2125253bf639deba9e32bb22f2abc799378c0d70fb98fd8108f54469
Oct 30 22:12:17.734637 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.734593    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"]
Oct 30 22:12:17.740568 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.740547    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.743409 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.743380    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s" event={"ID":"94aaca84-9a1a-45f3-921d-b48c4675bbec","Type":"ContainerStarted","Data":"e3d4d5ea2125253bf639deba9e32bb22f2abc799378c0d70fb98fd8108f54469"}
Oct 30 22:12:17.759248 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.759221    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"]
Oct 30 22:12:17.850533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.850492    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.850766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.850741    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.850948 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.850925    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmkd\" (UniqueName: \"kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.952236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.952150    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmkd\" (UniqueName: \"kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.952423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.952248    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.952423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.952333    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.953053 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.953026    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.954608 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.954580    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:17.971285 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:17.971255    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmkd\" (UniqueName: \"kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd\") pod \"maas-api-revoke-sub-cd2654d1-mccsn\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:18.061955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.061925    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:18.378662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.378634    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"]
Oct 30 22:12:18.437633 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:18.437586    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb053358_2e7c_4a86_b643_77308fa2200f.slice/crio-34455d274be2c70a774a330b68d667af2e88b7aec33e5a1df9bfed3b67427551 WatchSource:0}: Error finding container 34455d274be2c70a774a330b68d667af2e88b7aec33e5a1df9bfed3b67427551: Status 404 returned error can't find the container with id 34455d274be2c70a774a330b68d667af2e88b7aec33e5a1df9bfed3b67427551
Oct 30 22:12:18.655121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.655087    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:12:18.754715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.754678    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn" event={"ID":"fb053358-2e7c-4a86-b643-77308fa2200f","Type":"ContainerStarted","Data":"34455d274be2c70a774a330b68d667af2e88b7aec33e5a1df9bfed3b67427551"}
Oct 30 22:12:18.756371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.756341    6695 generic.go:356] "Generic (PLEG): container finished" podID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerID="645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25" exitCode=0
Oct 30 22:12:18.756504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.756375    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" event={"ID":"197c2517-8f1a-47b6-ac8a-d42ae83e4b56","Type":"ContainerDied","Data":"645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25"}
Oct 30 22:12:18.756504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.756388    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"
Oct 30 22:12:18.756504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.756438    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx" event={"ID":"197c2517-8f1a-47b6-ac8a-d42ae83e4b56","Type":"ContainerDied","Data":"49c268633a668f3f8006b10110abed423799db2c09f801a36c17393a0123fe41"}
Oct 30 22:12:18.756504 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.756472    6695 scope.go:122] "RemoveContainer" containerID="645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25"
Oct 30 22:12:18.758999 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.758976    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s" event={"ID":"94aaca84-9a1a-45f3-921d-b48c4675bbec","Type":"ContainerStarted","Data":"c5cb30ddb9fc24bcd62b9f59ec00fcf933e1a04804c1460b39e4a024a6d67e50"}
Oct 30 22:12:18.765790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.765759    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume\") pod \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") "
Oct 30 22:12:18.765945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.765929    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9\" (UniqueName: \"kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9\") pod \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\" (UID: \"197c2517-8f1a-47b6-ac8a-d42ae83e4b56\") "
Oct 30 22:12:18.766135 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.766113    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume" pod "197c2517-8f1a-47b6-ac8a-d42ae83e4b56" (UID: "197c2517-8f1a-47b6-ac8a-d42ae83e4b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:18.766595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.766574    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:18.768540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.768520    6695 scope.go:122] "RemoveContainer" containerID="645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25"
Oct 30 22:12:18.768836 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:18.768813    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25\": container with ID starting with 645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25 not found: ID does not exist" containerID="645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25"
Oct 30 22:12:18.768911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.768841    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25"} err="failed to get container status \"645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25\": rpc error: code = NotFound desc = could not find container \"645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25\": container with ID starting with 645d4d6038135eafa8b7cab42c47e49bbd82a797e87be8c6a71a41f16b875a25 not found: ID does not exist"
Oct 30 22:12:18.811337 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.811244    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9" pod "197c2517-8f1a-47b6-ac8a-d42ae83e4b56" (UID: "197c2517-8f1a-47b6-ac8a-d42ae83e4b56"). InnerVolumeSpecName "kube-api-access-277m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:18.820132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.820058    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s" podStartSLOduration=1.82004115 podStartE2EDuration="1.82004115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:18.818888634 +0000 UTC m=+3721.741346933" watchObservedRunningTime="2026-10-30 22:12:18.82004115 +0000 UTC m=+3721.742499448"
Oct 30 22:12:18.868168 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:18.868089    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-277m9\" (UniqueName: \"kubernetes.io/projected/197c2517-8f1a-47b6-ac8a-d42ae83e4b56-kube-api-access-277m9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:19.137905 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.137820    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"]
Oct 30 22:12:19.156336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.156302    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6537e6-7ffc5cb46c-72frx"]
Oct 30 22:12:19.216098 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.216065    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" path="/var/lib/kubelet/pods/197c2517-8f1a-47b6-ac8a-d42ae83e4b56/volumes"
Oct 30 22:12:19.769149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.769106    6695 generic.go:356] "Generic (PLEG): container finished" podID="fb053358-2e7c-4a86-b643-77308fa2200f" containerID="c177a03254c5672bfeb678354c97875575583daf24b2faf467395b3c069427fd" exitCode=0
Oct 30 22:12:19.769631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.769249    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn" event={"ID":"fb053358-2e7c-4a86-b643-77308fa2200f","Type":"ContainerDied","Data":"c177a03254c5672bfeb678354c97875575583daf24b2faf467395b3c069427fd"}
Oct 30 22:12:19.771879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.771854    6695 generic.go:356] "Generic (PLEG): container finished" podID="94aaca84-9a1a-45f3-921d-b48c4675bbec" containerID="c5cb30ddb9fc24bcd62b9f59ec00fcf933e1a04804c1460b39e4a024a6d67e50" exitCode=0
Oct 30 22:12:19.772006 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:19.771884    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s" event={"ID":"94aaca84-9a1a-45f3-921d-b48c4675bbec","Type":"ContainerDied","Data":"c5cb30ddb9fc24bcd62b9f59ec00fcf933e1a04804c1460b39e4a024a6d67e50"}
Oct 30 22:12:20.965039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:20.965008    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:20.993976 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:20.993946    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:21.096941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.096857    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs\" (UniqueName: \"kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs\") pod \"94aaca84-9a1a-45f3-921d-b48c4675bbec\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") "
Oct 30 22:12:21.096941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.096910    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca\") pod \"fb053358-2e7c-4a86-b643-77308fa2200f\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") "
Oct 30 22:12:21.096941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.096938    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca\") pod \"fb053358-2e7c-4a86-b643-77308fa2200f\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") "
Oct 30 22:12:21.097243 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.096986    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca\") pod \"94aaca84-9a1a-45f3-921d-b48c4675bbec\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") "
Oct 30 22:12:21.097315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.097274    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd\" (UniqueName: \"kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd\") pod \"fb053358-2e7c-4a86-b643-77308fa2200f\" (UID: \"fb053358-2e7c-4a86-b643-77308fa2200f\") "
Oct 30 22:12:21.097375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.097320    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca" pod "fb053358-2e7c-4a86-b643-77308fa2200f" (UID: "fb053358-2e7c-4a86-b643-77308fa2200f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:21.097444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.097370    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca\") pod \"94aaca84-9a1a-45f3-921d-b48c4675bbec\" (UID: \"94aaca84-9a1a-45f3-921d-b48c4675bbec\") "
Oct 30 22:12:21.097815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.097788    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca" pod "94aaca84-9a1a-45f3-921d-b48c4675bbec" (UID: "94aaca84-9a1a-45f3-921d-b48c4675bbec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:21.098123 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.098100    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:21.098200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.098130    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:21.099292 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.099265    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca" pod "fb053358-2e7c-4a86-b643-77308fa2200f" (UID: "fb053358-2e7c-4a86-b643-77308fa2200f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:21.099396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.099265    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca" pod "94aaca84-9a1a-45f3-921d-b48c4675bbec" (UID: "94aaca84-9a1a-45f3-921d-b48c4675bbec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:21.099396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.099367    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs" pod "94aaca84-9a1a-45f3-921d-b48c4675bbec" (UID: "94aaca84-9a1a-45f3-921d-b48c4675bbec"). InnerVolumeSpecName "kube-api-access-rkdgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:21.099476 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.099396    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd" pod "fb053358-2e7c-4a86-b643-77308fa2200f" (UID: "fb053358-2e7c-4a86-b643-77308fa2200f"). InnerVolumeSpecName "kube-api-access-kvmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:21.199639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.199571    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb053358-2e7c-4a86-b643-77308fa2200f-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:21.199639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.199632    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94aaca84-9a1a-45f3-921d-b48c4675bbec-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:21.199639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.199644    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmkd\" (UniqueName: \"kubernetes.io/projected/fb053358-2e7c-4a86-b643-77308fa2200f-kube-api-access-kvmkd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:21.199901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.199655    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkdgs\" (UniqueName: \"kubernetes.io/projected/94aaca84-9a1a-45f3-921d-b48c4675bbec-kube-api-access-rkdgs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:21.788857 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.788815    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"
Oct 30 22:12:21.789036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.788817    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s" event={"ID":"94aaca84-9a1a-45f3-921d-b48c4675bbec","Type":"ContainerDied","Data":"e3d4d5ea2125253bf639deba9e32bb22f2abc799378c0d70fb98fd8108f54469"}
Oct 30 22:12:21.789036 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.788972    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d4d5ea2125253bf639deba9e32bb22f2abc799378c0d70fb98fd8108f54469"
Oct 30 22:12:21.790457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.790435    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn"
Oct 30 22:12:21.790575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.790471    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd2654d1-mccsn" event={"ID":"fb053358-2e7c-4a86-b643-77308fa2200f","Type":"ContainerDied","Data":"34455d274be2c70a774a330b68d667af2e88b7aec33e5a1df9bfed3b67427551"}
Oct 30 22:12:21.790575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:21.790510    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34455d274be2c70a774a330b68d667af2e88b7aec33e5a1df9bfed3b67427551"
Oct 30 22:12:23.581889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:23.581854    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"]
Oct 30 22:12:23.583025 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:23.582992    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy" containerID="cri-o://de7070be8f8d80cd73db8b00d0dc6c76b0f7325d2a1f8f5dacfd7a521c3558f2" gracePeriod=30
Oct 30 22:12:24.181764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.181729    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"]
Oct 30 22:12:24.182443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182429    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fb053358-2e7c-4a86-b643-77308fa2200f" containerName="revoke-keys"
Oct 30 22:12:24.182492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182447    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb053358-2e7c-4a86-b643-77308fa2200f" containerName="revoke-keys"
Oct 30 22:12:24.182492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182466    6695 cpu_manager.go:414] "containerMap: removing container" podUID="94aaca84-9a1a-45f3-921d-b48c4675bbec" containerName="revoke-keys"
Oct 30 22:12:24.182492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182474    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94aaca84-9a1a-45f3-921d-b48c4675bbec" containerName="revoke-keys"
Oct 30 22:12:24.182639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182499    6695 cpu_manager.go:414] "containerMap: removing container" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerName="payload-pre-processing"
Oct 30 22:12:24.182639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182506    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerName="payload-pre-processing"
Oct 30 22:12:24.182639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182635    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb053358-2e7c-4a86-b643-77308fa2200f" containerName="revoke-keys"
Oct 30 22:12:24.182761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182649    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="94aaca84-9a1a-45f3-921d-b48c4675bbec" containerName="revoke-keys"
Oct 30 22:12:24.182761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.182658    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="197c2517-8f1a-47b6-ac8a-d42ae83e4b56" containerName="payload-pre-processing"
Oct 30 22:12:24.188878 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.188845    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.208509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.208465    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:24.208509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.208475    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:24.208509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.208481    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:12:24.208827 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.208526    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:24.210654 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.210625    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"]
Oct 30 22:12:24.357000 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.356965    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.357207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.357114    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klttt\" (UniqueName: \"kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.357288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.357202    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.458473 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.458383    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.458473 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.458467    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klttt\" (UniqueName: \"kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.458709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.458501    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.459148 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.459121    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.460914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.460881    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.481648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.481590    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klttt\" (UniqueName: \"kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.503863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.503829    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:24.815004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:24.814972    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"]
Oct 30 22:12:24.816438 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:24.816410    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab50af9_a1c4_4369_a086_ce20f9b48202.slice/crio-0ee87dca526dedc2eb8f9ba7ca16a2852733b3dcf5a59d31d4f975be7774ed42 WatchSource:0}: Error finding container 0ee87dca526dedc2eb8f9ba7ca16a2852733b3dcf5a59d31d4f975be7774ed42: Status 404 returned error can't find the container with id 0ee87dca526dedc2eb8f9ba7ca16a2852733b3dcf5a59d31d4f975be7774ed42
Oct 30 22:12:25.840334 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:25.840296    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls" event={"ID":"fab50af9-a1c4-4369-a086-ce20f9b48202","Type":"ContainerStarted","Data":"04ca62d225ca2acdc0498539dd3cb6efd788d95b21ef3aa0e4f0cbf9b201d1dd"}
Oct 30 22:12:25.840716 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:25.840397    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls" event={"ID":"fab50af9-a1c4-4369-a086-ce20f9b48202","Type":"ContainerStarted","Data":"0ee87dca526dedc2eb8f9ba7ca16a2852733b3dcf5a59d31d4f975be7774ed42"}
Oct 30 22:12:26.849838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:26.849802    6695 generic.go:356] "Generic (PLEG): container finished" podID="fab50af9-a1c4-4369-a086-ce20f9b48202" containerID="04ca62d225ca2acdc0498539dd3cb6efd788d95b21ef3aa0e4f0cbf9b201d1dd" exitCode=0
Oct 30 22:12:26.850266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:26.849872    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls" event={"ID":"fab50af9-a1c4-4369-a086-ce20f9b48202","Type":"ContainerDied","Data":"04ca62d225ca2acdc0498539dd3cb6efd788d95b21ef3aa0e4f0cbf9b201d1dd"}
Oct 30 22:12:27.038712 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.038681    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:27.200722 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.200597    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca\") pod \"fab50af9-a1c4-4369-a086-ce20f9b48202\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") "
Oct 30 22:12:27.201050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.200729    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca\") pod \"fab50af9-a1c4-4369-a086-ce20f9b48202\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") "
Oct 30 22:12:27.201050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.200759    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt\" (UniqueName: \"kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt\") pod \"fab50af9-a1c4-4369-a086-ce20f9b48202\" (UID: \"fab50af9-a1c4-4369-a086-ce20f9b48202\") "
Oct 30 22:12:27.201050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.201001    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca" pod "fab50af9-a1c4-4369-a086-ce20f9b48202" (UID: "fab50af9-a1c4-4369-a086-ce20f9b48202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:27.201295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.201273    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:27.203001 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.202972    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt" pod "fab50af9-a1c4-4369-a086-ce20f9b48202" (UID: "fab50af9-a1c4-4369-a086-ce20f9b48202"). InnerVolumeSpecName "kube-api-access-klttt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:27.203121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.203041    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca" pod "fab50af9-a1c4-4369-a086-ce20f9b48202" (UID: "fab50af9-a1c4-4369-a086-ce20f9b48202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:27.302395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.302358    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab50af9-a1c4-4369-a086-ce20f9b48202-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:27.302395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.302394    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klttt\" (UniqueName: \"kubernetes.io/projected/fab50af9-a1c4-4369-a086-ce20f9b48202-kube-api-access-klttt\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:27.532816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.532779    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"]
Oct 30 22:12:27.562363 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.562318    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"]
Oct 30 22:12:27.860126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.860034    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"
Oct 30 22:12:27.860540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.860041    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls" event={"ID":"fab50af9-a1c4-4369-a086-ce20f9b48202","Type":"ContainerDied","Data":"0ee87dca526dedc2eb8f9ba7ca16a2852733b3dcf5a59d31d4f975be7774ed42"}
Oct 30 22:12:27.860540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:27.860183    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee87dca526dedc2eb8f9ba7ca16a2852733b3dcf5a59d31d4f975be7774ed42"
Oct 30 22:12:28.872275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:28.872187    6695 generic.go:356] "Generic (PLEG): container finished" podID="79a33640-8558-4199-b50c-2c64140f338a" containerID="de7070be8f8d80cd73db8b00d0dc6c76b0f7325d2a1f8f5dacfd7a521c3558f2" exitCode=0
Oct 30 22:12:28.872728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:28.872278    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" event={"ID":"79a33640-8558-4199-b50c-2c64140f338a","Type":"ContainerDied","Data":"de7070be8f8d80cd73db8b00d0dc6c76b0f7325d2a1f8f5dacfd7a521c3558f2"}
Oct 30 22:12:28.872728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:28.872450    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh" event={"ID":"79a33640-8558-4199-b50c-2c64140f338a","Type":"ContainerDied","Data":"48a6b4a8da0f72f258a99f1fb74f73c22ec55c5ab74eefb392d82b1ab9c1295d"}
Oct 30 22:12:28.872728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:28.872491    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a6b4a8da0f72f258a99f1fb74f73c22ec55c5ab74eefb392d82b1ab9c1295d"
Oct 30 22:12:28.872728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:28.872510    6695 scope.go:122] "RemoveContainer" containerID="7896cbd2b09f2af560736ccc12171e0ed7c021e02a97c512993ff0badf5d87dc"
Oct 30 22:12:28.873250 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:28.872286    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:12:29.048697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.048594    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.048901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.048711    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.048901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.048771    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.048901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.048832    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.048901 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.048892    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.049154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.048995    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:29.049154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049019    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.049284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049175    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:29.049284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049180    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.049284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049215    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.049472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049293    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert\") pod \"79a33640-8558-4199-b50c-2c64140f338a\" (UID: \"79a33640-8558-4199-b50c-2c64140f338a\") "
Oct 30 22:12:29.049635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049459    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:29.049635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049470    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:29.049893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049868    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:29.050010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.049988    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.050010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.050006    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.050211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.050023    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a33640-8558-4199-b50c-2c64140f338a-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.050211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.050038    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.050211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.050048    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.051145 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.051124    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:12:29.051229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.051174    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:29.051442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.051420    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:29.051495 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.051442    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8" pod "79a33640-8558-4199-b50c-2c64140f338a" (UID: "79a33640-8558-4199-b50c-2c64140f338a"). InnerVolumeSpecName "kube-api-access-k6hz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:29.151117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.151029    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.151117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.151063    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a33640-8558-4199-b50c-2c64140f338a-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.151117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.151073    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a33640-8558-4199-b50c-2c64140f338a-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.151117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.151082    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6hz8\" (UniqueName: \"kubernetes.io/projected/79a33640-8558-4199-b50c-2c64140f338a-kube-api-access-k6hz8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:29.385530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.385495    6695 scope.go:122] "RemoveContainer" containerID="9856eb75f097f945f0019f79ad8bc737b428cb488aecf9d8429937b753072613"
Oct 30 22:12:29.395986 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.395964    6695 scope.go:122] "RemoveContainer" containerID="04b85b1c703d2e49425e687a3ca167ef85a368c502325c2622b28dadeac7a18b"
Oct 30 22:12:29.405482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.405461    6695 scope.go:122] "RemoveContainer" containerID="48dbaf4fe7f2f9e294a7304c2443d225ec2cfca93c544b89f57eb3cb2eec4176"
Oct 30 22:12:29.881542 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.881511    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"
Oct 30 22:12:29.975812 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.975773    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"]
Oct 30 22:12:29.979983 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:29.979947    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6537e6-openshift-default-6f4ccbc885-pq5bh"]
Oct 30 22:12:31.031967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.031930    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"]
Oct 30 22:12:31.043246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.043198    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"]
Oct 30 22:12:31.046857 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.046823    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67824d27-fhg8p"]
Oct 30 22:12:31.050412 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.050378    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a6d34d5-m25r2"]
Oct 30 22:12:31.076323 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:31.076288    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:12:31.215160 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.215128    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297ec5e3-ffe8-4638-b2e5-db493c895abf" path="/var/lib/kubelet/pods/297ec5e3-ffe8-4638-b2e5-db493c895abf/volumes"
Oct 30 22:12:31.215663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.215648    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a33640-8558-4199-b50c-2c64140f338a" path="/var/lib/kubelet/pods/79a33640-8558-4199-b50c-2c64140f338a/volumes"
Oct 30 22:12:31.216077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:31.216064    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0f7459-61a7-4679-b0a9-3b9059b96413" path="/var/lib/kubelet/pods/ab0f7459-61a7-4679-b0a9-3b9059b96413/volumes"
Oct 30 22:12:32.207580 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.207542    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8" podUID="1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" containerName="main" containerID="cri-o://99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3" gracePeriod=45
Oct 30 22:12:32.511748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.511720    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:12:32.533293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.533257    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl" podUID="a990936b-267c-4d6d-8a45-9728ca2d4e7d" containerName="payload-processing" containerID="cri-o://3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e" gracePeriod=25
Oct 30 22:12:32.563143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.563100    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerName="payload-pre-processing" containerID="cri-o://6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e" gracePeriod=25
Oct 30 22:12:32.597740 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.597704    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir\") pod \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") "
Oct 30 22:12:32.597937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.597772    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4\" (UniqueName: \"kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4\") pod \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") "
Oct 30 22:12:32.597937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.597843    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home\") pod \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") "
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.597986    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs\") pod \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") "
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598040    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir" pod "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" (UID: "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598068    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache\") pod \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") "
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598097    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm\") pod \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\" (UID: \"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a\") "
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598276    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home" pod "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" (UID: "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598556    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache" pod "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" (UID: "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598765    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598789    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.598802    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.601032    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs" pod "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" (UID: "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.602974    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm" pod "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" (UID: "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:32.603807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.603100    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4" pod "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" (UID: "1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a"). InnerVolumeSpecName "kube-api-access-l5sz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:32.700071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.700036    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5sz4\" (UniqueName: \"kubernetes.io/projected/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-kube-api-access-l5sz4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:32.700071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.700073    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:32.700415 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.700091    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:32.742059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.741957    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"]
Oct 30 22:12:32.806414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.806385    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:12:32.915031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.914854    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume\") pod \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") "
Oct 30 22:12:32.915031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.914924    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m\" (UniqueName: \"kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m\") pod \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\" (UID: \"a990936b-267c-4d6d-8a45-9728ca2d4e7d\") "
Oct 30 22:12:32.915704 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.915674    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume" pod "a990936b-267c-4d6d-8a45-9728ca2d4e7d" (UID: "a990936b-267c-4d6d-8a45-9728ca2d4e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:32.918967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.918931    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m" pod "a990936b-267c-4d6d-8a45-9728ca2d4e7d" (UID: "a990936b-267c-4d6d-8a45-9728ca2d4e7d"). InnerVolumeSpecName "kube-api-access-k8c2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:32.925879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.925823    6695 generic.go:356] "Generic (PLEG): container finished" podID="1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" containerID="99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3" exitCode=0
Oct 30 22:12:32.926082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.925945    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8" event={"ID":"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a","Type":"ContainerDied","Data":"99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3"}
Oct 30 22:12:32.926082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.925981    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8" event={"ID":"1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a","Type":"ContainerDied","Data":"52c015654ca891810cacdd305dadb31f1a6a0f37d8626d1932c1cd6e376ee414"}
Oct 30 22:12:32.926082 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.926004    6695 scope.go:122] "RemoveContainer" containerID="99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3"
Oct 30 22:12:32.926248 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.926144    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"
Oct 30 22:12:32.930903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.930877    6695 generic.go:356] "Generic (PLEG): container finished" podID="a990936b-267c-4d6d-8a45-9728ca2d4e7d" containerID="3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e" exitCode=0
Oct 30 22:12:32.931092 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.931075    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl" event={"ID":"a990936b-267c-4d6d-8a45-9728ca2d4e7d","Type":"ContainerDied","Data":"3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e"}
Oct 30 22:12:32.931151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.931118    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl" event={"ID":"a990936b-267c-4d6d-8a45-9728ca2d4e7d","Type":"ContainerDied","Data":"beaeff6370d5fc77029c720090127cea032a0522d4971cb787453f10bb6bae99"}
Oct 30 22:12:32.931200 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.931171    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"
Oct 30 22:12:32.943762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.943736    6695 scope.go:122] "RemoveContainer" containerID="99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3"
Oct 30 22:12:32.944133 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:32.944111    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3\": container with ID starting with 99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3 not found: ID does not exist" containerID="99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3"
Oct 30 22:12:32.944233 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.944144    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3"} err="failed to get container status \"99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3\": rpc error: code = NotFound desc = could not find container \"99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3\": container with ID starting with 99fa7cd2e15f0757832883caf30c5ef2b46c57c5b3c07e63160d842a7ea47dc3 not found: ID does not exist"
Oct 30 22:12:32.944233 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.944165    6695 scope.go:122] "RemoveContainer" containerID="3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e"
Oct 30 22:12:32.956774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.955906    6695 scope.go:122] "RemoveContainer" containerID="3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e"
Oct 30 22:12:32.957942 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:32.957047    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e\": container with ID starting with 3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e not found: ID does not exist" containerID="3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e"
Oct 30 22:12:32.957942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.957086    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e"} err="failed to get container status \"3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e\": rpc error: code = NotFound desc = could not find container \"3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e\": container with ID starting with 3dac2e5636947b061bbc8c7302a6900a1b2e148aa34a612f3e4c0203a737820e not found: ID does not exist"
Oct 30 22:12:32.984738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.984699    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"]
Oct 30 22:12:32.989515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.989483    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"]
Oct 30 22:12:32.990188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990172    6695 cpu_manager.go:414] "containerMap: removing container" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy"
Oct 30 22:12:32.990235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990190    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy"
Oct 30 22:12:32.990235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990204    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a990936b-267c-4d6d-8a45-9728ca2d4e7d" containerName="payload-processing"
Oct 30 22:12:32.990235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990209    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a990936b-267c-4d6d-8a45-9728ca2d4e7d" containerName="payload-processing"
Oct 30 22:12:32.990235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990221    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fab50af9-a1c4-4369-a086-ce20f9b48202" containerName="revoke-keys"
Oct 30 22:12:32.990235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990226    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab50af9-a1c4-4369-a086-ce20f9b48202" containerName="revoke-keys"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990239    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" containerName="main"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990244    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" containerName="main"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990250    6695 cpu_manager.go:414] "containerMap: removing container" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990255    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990377    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990387    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a990936b-267c-4d6d-8a45-9728ca2d4e7d" containerName="payload-processing"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990394    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab50af9-a1c4-4369-a086-ce20f9b48202" containerName="revoke-keys"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990402    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" containerName="main"
Oct 30 22:12:32.990431 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.990408    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a33640-8558-4199-b50c-2c64140f338a" containerName="istio-proxy"
Oct 30 22:12:32.995793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.995673    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:32.996449 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:32.996423    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a42c31/test-model-a42c31-kserve-5dd47ccbc5-hphr8"]
Oct 30 22:12:33.011210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.011176    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:33.011406 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.011178    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:12:33.011406 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.011313    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"]
Oct 30 22:12:33.018479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.018438    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a990936b-267c-4d6d-8a45-9728ca2d4e7d-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:33.018479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.018481    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8c2m\" (UniqueName: \"kubernetes.io/projected/a990936b-267c-4d6d-8a45-9728ca2d4e7d-kube-api-access-k8c2m\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:33.033324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.033287    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"]
Oct 30 22:12:33.038281 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.038248    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5454af-59d8c5b66c-ddrtl"]
Oct 30 22:12:33.085187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.085153    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:33.085187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.085187    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:33.120132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.120094    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.120315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.120299    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmlf\" (UniqueName: \"kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.120386 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.120331    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.215125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.215082    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a" path="/var/lib/kubelet/pods/1fe77f2d-c2a6-4bff-9ee2-724ac5be6f1a/volumes"
Oct 30 22:12:33.215551 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.215536    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a990936b-267c-4d6d-8a45-9728ca2d4e7d" path="/var/lib/kubelet/pods/a990936b-267c-4d6d-8a45-9728ca2d4e7d/volumes"
Oct 30 22:12:33.221181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.221146    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmlf\" (UniqueName: \"kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.221381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.221201    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.221381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.221274    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.221918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.221894    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.223565 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.223546    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.240699 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.240666    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmlf\" (UniqueName: \"kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf\") pod \"maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.317447 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.317409    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:33.632588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.632550    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"]
Oct 30 22:12:33.635201 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:33.635164    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f09d56c_bb9a_4b1a_8ac7_78fb785f8de1.slice/crio-f0792b48dde16c6730437160e4b257776546d873c3c95541acf752b0de29fd88 WatchSource:0}: Error finding container f0792b48dde16c6730437160e4b257776546d873c3c95541acf752b0de29fd88: Status 404 returned error can't find the container with id f0792b48dde16c6730437160e4b257776546d873c3c95541acf752b0de29fd88
Oct 30 22:12:33.943413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.943378    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds" event={"ID":"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1","Type":"ContainerStarted","Data":"81c3ceb61c38a3a04f3367c31683381eaff7e577b688020d0ffdf46859bf2cac"}
Oct 30 22:12:33.943586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:33.943429    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds" event={"ID":"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1","Type":"ContainerStarted","Data":"f0792b48dde16c6730437160e4b257776546d873c3c95541acf752b0de29fd88"}
Oct 30 22:12:34.011321 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:34.011254    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds" podStartSLOduration=2.011242068 podStartE2EDuration="2.011242068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:34.009811531 +0000 UTC m=+3736.932269829" watchObservedRunningTime="2026-10-30 22:12:34.011242068 +0000 UTC m=+3736.933700365"
Oct 30 22:12:34.642939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:34.642903    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"]
Oct 30 22:12:34.672424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:34.672388    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"]
Oct 30 22:12:34.953738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:34.953577    6695 generic.go:356] "Generic (PLEG): container finished" podID="2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" containerID="81c3ceb61c38a3a04f3367c31683381eaff7e577b688020d0ffdf46859bf2cac" exitCode=0
Oct 30 22:12:34.953738 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:34.953657    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds" event={"ID":"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1","Type":"ContainerDied","Data":"81c3ceb61c38a3a04f3367c31683381eaff7e577b688020d0ffdf46859bf2cac"}
Oct 30 22:12:36.112076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.112040    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:36.162100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.162067    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca\") pod \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") "
Oct 30 22:12:36.162299 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.162122    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca\") pod \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") "
Oct 30 22:12:36.162380 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.162312    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf\" (UniqueName: \"kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf\") pod \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\" (UID: \"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1\") "
Oct 30 22:12:36.162517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.162491    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca" pod "2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" (UID: "2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:36.162791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.162770    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:36.164749 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.164724    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf" pod "2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" (UID: "2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1"). InnerVolumeSpecName "kube-api-access-rnmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:36.169862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.169819    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca" pod "2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" (UID: "2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:36.264510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.264416    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmlf\" (UniqueName: \"kubernetes.io/projected/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-kube-api-access-rnmlf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:36.264510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.264455    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:36.971230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.971199    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"
Oct 30 22:12:36.971230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.971224    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds" event={"ID":"2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1","Type":"ContainerDied","Data":"f0792b48dde16c6730437160e4b257776546d873c3c95541acf752b0de29fd88"}
Oct 30 22:12:36.971467 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:36.971274    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0792b48dde16c6730437160e4b257776546d873c3c95541acf752b0de29fd88"
Oct 30 22:12:38.032212 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:38.032170    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"]
Oct 30 22:12:38.042831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:38.042797    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517f28b1-fwxzk"]
Oct 30 22:12:39.139442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:39.139395    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:39.214776 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:39.214745    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f38dca9-a36a-400b-b5a8-56c085719157" path="/var/lib/kubelet/pods/4f38dca9-a36a-400b-b5a8-56c085719157/volumes"
Oct 30 22:12:39.644141 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:39.644072    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerName="payload-processing" containerID="cri-o://f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed" gracePeriod=25
Oct 30 22:12:39.672751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:39.672710    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" containerName="payload-pre-processing" containerID="cri-o://cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a" gracePeriod=25
Oct 30 22:12:41.629817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:41.629769    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:42.643042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:42.643007    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:43.447169 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:43.447136    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"]
Oct 30 22:12:46.279340 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:46.279296    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"]
Oct 30 22:12:46.311822 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:46.311779    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19461-z8mls"]
Oct 30 22:12:47.217128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:47.217093    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab50af9-a1c4-4369-a086-ce20f9b48202" path="/var/lib/kubelet/pods/fab50af9-a1c4-4369-a086-ce20f9b48202/volumes"
Oct 30 22:12:47.769128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:47.769085    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj" podUID="5edf70a6-ef3f-4ca3-93c5-d3219422a139" containerName="main" containerID="cri-o://ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8" gracePeriod=45
Oct 30 22:12:48.061407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.061379    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:12:48.137517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.135467    6695 generic.go:356] "Generic (PLEG): container finished" podID="5edf70a6-ef3f-4ca3-93c5-d3219422a139" containerID="ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8" exitCode=0
Oct 30 22:12:48.137517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.135813    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj" event={"ID":"5edf70a6-ef3f-4ca3-93c5-d3219422a139","Type":"ContainerDied","Data":"ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8"}
Oct 30 22:12:48.137517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.135871    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj" event={"ID":"5edf70a6-ef3f-4ca3-93c5-d3219422a139","Type":"ContainerDied","Data":"7463bea41fceaa7b408110ef233a4a354b66ea6bf2afca26b9cd9327b1275d04"}
Oct 30 22:12:48.137517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.135894    6695 scope.go:122] "RemoveContainer" containerID="ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8"
Oct 30 22:12:48.137517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.136068    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"
Oct 30 22:12:48.184736 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.184712    6695 scope.go:122] "RemoveContainer" containerID="ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8"
Oct 30 22:12:48.185084 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:48.185062    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8\": container with ID starting with ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8 not found: ID does not exist" containerID="ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8"
Oct 30 22:12:48.185138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.185095    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8"} err="failed to get container status \"ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8\": rpc error: code = NotFound desc = could not find container \"ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8\": container with ID starting with ec8bfd1ba976bf576f3f9e47303fd1b7c44269b0d485b2cda4fd10a0178eaad8 not found: ID does not exist"
Oct 30 22:12:48.244725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.244691    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv\" (UniqueName: \"kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv\") pod \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") "
Oct 30 22:12:48.244926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.244745    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir\") pod \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") "
Oct 30 22:12:48.244926 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.244786    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs\" (UniqueName: \"kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs\") pod \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") "
Oct 30 22:12:48.245046 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.244921    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache\") pod \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") "
Oct 30 22:12:48.245104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245031    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir" pod "5edf70a6-ef3f-4ca3-93c5-d3219422a139" (UID: "5edf70a6-ef3f-4ca3-93c5-d3219422a139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:48.245104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245048    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm\") pod \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") "
Oct 30 22:12:48.245104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245077    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home\") pod \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\" (UID: \"5edf70a6-ef3f-4ca3-93c5-d3219422a139\") "
Oct 30 22:12:48.245309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245200    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache" pod "5edf70a6-ef3f-4ca3-93c5-d3219422a139" (UID: "5edf70a6-ef3f-4ca3-93c5-d3219422a139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:48.245418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245394    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home" pod "5edf70a6-ef3f-4ca3-93c5-d3219422a139" (UID: "5edf70a6-ef3f-4ca3-93c5-d3219422a139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:48.245641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245619    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.245727 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245648    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.245727 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.245661    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.247161 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.247140    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm" pod "5edf70a6-ef3f-4ca3-93c5-d3219422a139" (UID: "5edf70a6-ef3f-4ca3-93c5-d3219422a139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:48.247376 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.247349    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs" pod "5edf70a6-ef3f-4ca3-93c5-d3219422a139" (UID: "5edf70a6-ef3f-4ca3-93c5-d3219422a139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:48.247478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.247383    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv" pod "5edf70a6-ef3f-4ca3-93c5-d3219422a139" (UID: "5edf70a6-ef3f-4ca3-93c5-d3219422a139"). InnerVolumeSpecName "kube-api-access-xvpzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:48.348086 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.348048    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5edf70a6-ef3f-4ca3-93c5-d3219422a139-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.348086 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.348086    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvpzv\" (UniqueName: \"kubernetes.io/projected/5edf70a6-ef3f-4ca3-93c5-d3219422a139-kube-api-access-xvpzv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.348379 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.348103    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5edf70a6-ef3f-4ca3-93c5-d3219422a139-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.448196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.448141    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerName="payload-processing" containerID="cri-o://02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8" gracePeriod=25
Oct 30 22:12:48.499273 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.499222    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"]
Oct 30 22:12:48.503267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.503244    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-19a725/test-model-19a725-kserve-7fbbcc5785-sgbfj"]
Oct 30 22:12:48.786355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.786310    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"]
Oct 30 22:12:48.791795 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:48.791755    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a42c31-cvxds"]
Oct 30 22:12:49.046192 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:49.046157    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"]
Oct 30 22:12:49.050791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:49.050761    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ae56c94-h28s9"]
Oct 30 22:12:49.138821 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:49.138783    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:49.215364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:49.215331    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" path="/var/lib/kubelet/pods/2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1/volumes"
Oct 30 22:12:49.215839 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:49.215823    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34edc899-f332-4e5a-97ea-d0214e1abe69" path="/var/lib/kubelet/pods/34edc899-f332-4e5a-97ea-d0214e1abe69/volumes"
Oct 30 22:12:49.216239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:49.216227    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edf70a6-ef3f-4ca3-93c5-d3219422a139" path="/var/lib/kubelet/pods/5edf70a6-ef3f-4ca3-93c5-d3219422a139/volumes"
Oct 30 22:12:50.993998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:50.993951    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"]
Oct 30 22:12:51.005871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:51.005836    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5454af-l9x2s"]
Oct 30 22:12:51.214987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:51.214955    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94aaca84-9a1a-45f3-921d-b48c4675bbec" path="/var/lib/kubelet/pods/94aaca84-9a1a-45f3-921d-b48c4675bbec/volumes"
Oct 30 22:12:51.630061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:51.630022    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:51.827413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:51.827371    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:52.644510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.644100    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:52.846788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.846760    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:12:52.942035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.941993    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume\") pod \"a8e019eb-f65a-47af-afb9-5ec631bd285c\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") "
Oct 30 22:12:52.942219 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.942124    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x\" (UniqueName: \"kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x\") pod \"a8e019eb-f65a-47af-afb9-5ec631bd285c\" (UID: \"a8e019eb-f65a-47af-afb9-5ec631bd285c\") "
Oct 30 22:12:52.942465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.942439    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume" pod "a8e019eb-f65a-47af-afb9-5ec631bd285c" (UID: "a8e019eb-f65a-47af-afb9-5ec631bd285c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:52.942683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.942665    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e019eb-f65a-47af-afb9-5ec631bd285c-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:52.944290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:52.944269    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x" pod "a8e019eb-f65a-47af-afb9-5ec631bd285c" (UID: "a8e019eb-f65a-47af-afb9-5ec631bd285c"). InnerVolumeSpecName "kube-api-access-x676x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:53.043700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.043668    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x676x\" (UniqueName: \"kubernetes.io/projected/a8e019eb-f65a-47af-afb9-5ec631bd285c-kube-api-access-x676x\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:53.177030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.176938    6695 generic.go:356] "Generic (PLEG): container finished" podID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerID="6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e" exitCode=0
Oct 30 22:12:53.177030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.176984    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"
Oct 30 22:12:53.177257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.177028    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" event={"ID":"a8e019eb-f65a-47af-afb9-5ec631bd285c","Type":"ContainerDied","Data":"6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e"}
Oct 30 22:12:53.177257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.177082    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs" event={"ID":"a8e019eb-f65a-47af-afb9-5ec631bd285c","Type":"ContainerDied","Data":"7b24564bb6e5171c1ce9151af859c34af86457b8d821dfd514cd33351b8aad8e"}
Oct 30 22:12:53.177257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.177104    6695 scope.go:122] "RemoveContainer" containerID="6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e"
Oct 30 22:12:53.188026 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.188010    6695 scope.go:122] "RemoveContainer" containerID="6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e"
Oct 30 22:12:53.188286 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:53.188269    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e\": container with ID starting with 6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e not found: ID does not exist" containerID="6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e"
Oct 30 22:12:53.188336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.188293    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e"} err="failed to get container status \"6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e\": rpc error: code = NotFound desc = could not find container \"6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e\": container with ID starting with 6a8fc22e17bccba815762f84524a6a21350c60bf32bdf3fda261eca6b7c2b70e not found: ID does not exist"
Oct 30 22:12:53.229058 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.229011    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"]
Oct 30 22:12:53.239985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.239955    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5454af-5c8ccfbb47-pr8qs"]
Oct 30 22:12:53.876929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.876898    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"]
Oct 30 22:12:53.877357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:53.877096    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" containerID="cri-o://2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb" gracePeriod=30
Oct 30 22:12:54.421691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.420960    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"]
Oct 30 22:12:54.422530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.422495    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerName="payload-pre-processing"
Oct 30 22:12:54.422766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.422742    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerName="payload-pre-processing"
Oct 30 22:12:54.422941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.422917    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" containerName="revoke-keys"
Oct 30 22:12:54.423079 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.423066    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" containerName="revoke-keys"
Oct 30 22:12:54.423232 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.423220    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5edf70a6-ef3f-4ca3-93c5-d3219422a139" containerName="main"
Oct 30 22:12:54.423345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.423333    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edf70a6-ef3f-4ca3-93c5-d3219422a139" containerName="main"
Oct 30 22:12:54.423882 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.423863    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" containerName="payload-pre-processing"
Oct 30 22:12:54.424038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.424024    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f09d56c-bb9a-4b1a-8ac7-78fb785f8de1" containerName="revoke-keys"
Oct 30 22:12:54.424195 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.424178    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edf70a6-ef3f-4ca3-93c5-d3219422a139" containerName="main"
Oct 30 22:12:54.438220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.430580    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.438220 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.432728    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"]
Oct 30 22:12:54.449254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.449222    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:54.449384 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.449366    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:54.465181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.465141    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffl6\" (UniqueName: \"kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.465364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.465185    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.465364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.465211    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.512214 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.512183    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:12:54.512214 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.512183    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:54.567218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.567184    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffl6\" (UniqueName: \"kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.567218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.567219    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.567488 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.567237    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.567912 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.567879    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.569573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.569552    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.586538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.586513    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffl6\" (UniqueName: \"kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6\") pod \"maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.753775 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.753693    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:54.790207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.790169    6695 patch_prober.go:29] interesting pod/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:12:54.790366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:54.790238    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:12:55.094934 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:55.094900    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"]
Oct 30 22:12:55.098992 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:55.098960    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode832671e_ed97_4170_9997_26bcda01fec3.slice/crio-8eebdf25e18f307ba63730547f280897d4a585b792b2d383cb6031191db5c605 WatchSource:0}: Error finding container 8eebdf25e18f307ba63730547f280897d4a585b792b2d383cb6031191db5c605: Status 404 returned error can't find the container with id 8eebdf25e18f307ba63730547f280897d4a585b792b2d383cb6031191db5c605
Oct 30 22:12:55.194277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:55.194239    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp" event={"ID":"e832671e-ed97-4170-9997-26bcda01fec3","Type":"ContainerStarted","Data":"8eebdf25e18f307ba63730547f280897d4a585b792b2d383cb6031191db5c605"}
Oct 30 22:12:55.217328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:55.217292    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e019eb-f65a-47af-afb9-5ec631bd285c" path="/var/lib/kubelet/pods/a8e019eb-f65a-47af-afb9-5ec631bd285c/volumes"
Oct 30 22:12:56.217520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.217481    6695 generic.go:356] "Generic (PLEG): container finished" podID="e832671e-ed97-4170-9997-26bcda01fec3" containerID="1115c6e9044a0ad5d01fa47ca686b455ea237b2bcf704a6b12f7c449a98b1cbf" exitCode=0
Oct 30 22:12:56.218070 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.217557    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp" event={"ID":"e832671e-ed97-4170-9997-26bcda01fec3","Type":"ContainerDied","Data":"1115c6e9044a0ad5d01fa47ca686b455ea237b2bcf704a6b12f7c449a98b1cbf"}
Oct 30 22:12:56.487698 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.487661    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"]
Oct 30 22:12:56.492247 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.492224    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.510956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.510920    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"]
Oct 30 22:12:56.616398 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.616365    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.616624 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.616507    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.616717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.616624    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dv5\" (UniqueName: \"kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.718227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.718186    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.718433 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.718269    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dv5\" (UniqueName: \"kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.718482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.718464    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.719166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.719143    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.720668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.720643    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.738174 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.738095    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dv5\" (UniqueName: \"kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5\") pod \"maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:56.806547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:56.806516    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:57.121305 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.121270    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"]
Oct 30 22:12:57.124457 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:57.124424    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2da20a6_beb9_4e28_afb8_c6a04a7ed20e.slice/crio-85e6f0b306e9428402f4f36c9e511368ed59ce6c3dabe71939037f63f56f5180 WatchSource:0}: Error finding container 85e6f0b306e9428402f4f36c9e511368ed59ce6c3dabe71939037f63f56f5180: Status 404 returned error can't find the container with id 85e6f0b306e9428402f4f36c9e511368ed59ce6c3dabe71939037f63f56f5180
Oct 30 22:12:57.227400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.227366    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h" event={"ID":"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e","Type":"ContainerStarted","Data":"85e6f0b306e9428402f4f36c9e511368ed59ce6c3dabe71939037f63f56f5180"}
Oct 30 22:12:57.361163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.361138    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:57.428488 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.428386    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca\") pod \"e832671e-ed97-4170-9997-26bcda01fec3\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") "
Oct 30 22:12:57.428709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.428654    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca\") pod \"e832671e-ed97-4170-9997-26bcda01fec3\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") "
Oct 30 22:12:57.428817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.428788    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6\" (UniqueName: \"kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6\") pod \"e832671e-ed97-4170-9997-26bcda01fec3\" (UID: \"e832671e-ed97-4170-9997-26bcda01fec3\") "
Oct 30 22:12:57.428817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.428796    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca" pod "e832671e-ed97-4170-9997-26bcda01fec3" (UID: "e832671e-ed97-4170-9997-26bcda01fec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:57.429338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.429307    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:57.431103 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.431080    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6" pod "e832671e-ed97-4170-9997-26bcda01fec3" (UID: "e832671e-ed97-4170-9997-26bcda01fec3"). InnerVolumeSpecName "kube-api-access-rffl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:57.431379 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.431344    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca" pod "e832671e-ed97-4170-9997-26bcda01fec3" (UID: "e832671e-ed97-4170-9997-26bcda01fec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:57.530726 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.530694    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e832671e-ed97-4170-9997-26bcda01fec3-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:57.530726 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:57.530725    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rffl6\" (UniqueName: \"kubernetes.io/projected/e832671e-ed97-4170-9997-26bcda01fec3-kube-api-access-rffl6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:58.019058 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.018968    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"]
Oct 30 22:12:58.019241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.019191    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy" containerID="cri-o://7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550" gracePeriod=30
Oct 30 22:12:58.235757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.235724    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"
Oct 30 22:12:58.235757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.235745    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp" event={"ID":"e832671e-ed97-4170-9997-26bcda01fec3","Type":"ContainerDied","Data":"8eebdf25e18f307ba63730547f280897d4a585b792b2d383cb6031191db5c605"}
Oct 30 22:12:58.236235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.235797    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eebdf25e18f307ba63730547f280897d4a585b792b2d383cb6031191db5c605"
Oct 30 22:12:58.237447 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.237420    6695 generic.go:356] "Generic (PLEG): container finished" podID="c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" containerID="994bcc47ff2f2dbdfb91ade2445e5b8dbec812e39f28fe68052a37e2debdc908" exitCode=0
Oct 30 22:12:58.237571 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.237500    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h" event={"ID":"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e","Type":"ContainerDied","Data":"994bcc47ff2f2dbdfb91ade2445e5b8dbec812e39f28fe68052a37e2debdc908"}
Oct 30 22:12:58.576592 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.576556    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"]
Oct 30 22:12:58.577446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.577424    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e832671e-ed97-4170-9997-26bcda01fec3" containerName="revoke-keys"
Oct 30 22:12:58.577446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.577447    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e832671e-ed97-4170-9997-26bcda01fec3" containerName="revoke-keys"
Oct 30 22:12:58.577596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.577556    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e832671e-ed97-4170-9997-26bcda01fec3" containerName="revoke-keys"
Oct 30 22:12:58.581076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.581058    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.599692 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.599664    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"]
Oct 30 22:12:58.646844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.646813    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.647010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.646852    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.647010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.646977    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdl86\" (UniqueName: \"kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.748710 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.748670    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdl86\" (UniqueName: \"kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.748902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.748883    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.748951 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.748916    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.749723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.749692    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.751691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.751662    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.768034 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.768004    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdl86\" (UniqueName: \"kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86\") pod \"maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:58.895818 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:58.895787    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:12:59.145691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.145624    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 22:12:59.203964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.203932    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"]
Oct 30 22:12:59.205965 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:12:59.205930    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28ee9c0_ef7e_47c3_b9b6_e44a4f427f3a.slice/crio-7b486676c9e7e7219d1c2834fdbcad08166671a345cd229bd7a55ae2c135273c WatchSource:0}: Error finding container 7b486676c9e7e7219d1c2834fdbcad08166671a345cd229bd7a55ae2c135273c: Status 404 returned error can't find the container with id 7b486676c9e7e7219d1c2834fdbcad08166671a345cd229bd7a55ae2c135273c
Oct 30 22:12:59.248189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.248161    6695 generic.go:356] "Generic (PLEG): container finished" podID="d0b41768-b63b-44df-94a1-292629eb7d29" containerID="2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb" exitCode=0
Oct 30 22:12:59.248544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.248208    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"
Oct 30 22:12:59.248544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.248239    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" event={"ID":"d0b41768-b63b-44df-94a1-292629eb7d29","Type":"ContainerDied","Data":"2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb"}
Oct 30 22:12:59.248544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.248291    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm" event={"ID":"d0b41768-b63b-44df-94a1-292629eb7d29","Type":"ContainerDied","Data":"ae13773d75b645a18c0c2bee48e627a4dead7e191c2428794afe4facfdbf2ebd"}
Oct 30 22:12:59.248544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.248314    6695 scope.go:122] "RemoveContainer" containerID="2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb"
Oct 30 22:12:59.249890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.249868    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c" event={"ID":"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a","Type":"ContainerStarted","Data":"7b486676c9e7e7219d1c2834fdbcad08166671a345cd229bd7a55ae2c135273c"}
Oct 30 22:12:59.254780 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.254756    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.254864 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.254853    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.254932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.254915    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.254962    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255051 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255042    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255060    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:59.255173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255087    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255124    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255171    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:59.255322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255194    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:59.255322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255266    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255316    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data\") pod \"d0b41768-b63b-44df-94a1-292629eb7d29\" (UID: \"d0b41768-b63b-44df-94a1-292629eb7d29\") "
Oct 30 22:12:59.255855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255508    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:59.255855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255647    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:59.255855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255812    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.255855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255828    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.255855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255841    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.255855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255855    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.256228 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.255870    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b41768-b63b-44df-94a1-292629eb7d29-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.257550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.257522    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "kube-api-access-57pnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:59.257683 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.257658    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:59.258161 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.258131    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:12:59.258300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.258257    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy" pod "d0b41768-b63b-44df-94a1-292629eb7d29" (UID: "d0b41768-b63b-44df-94a1-292629eb7d29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:59.260121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.260096    6695 scope.go:122] "RemoveContainer" containerID="7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"
Oct 30 22:12:59.270649 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.270625    6695 scope.go:122] "RemoveContainer" containerID="2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb"
Oct 30 22:12:59.270959 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:59.270933    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb\": container with ID starting with 2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb not found: ID does not exist" containerID="2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb"
Oct 30 22:12:59.271031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.270968    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb"} err="failed to get container status \"2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb\": rpc error: code = NotFound desc = could not find container \"2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb\": container with ID starting with 2be68b45bb0b3f2fa12b3d359acd973960e9bb36edf8b595236eba4e40ea7cdb not found: ID does not exist"
Oct 30 22:12:59.271031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.270986    6695 scope.go:122] "RemoveContainer" containerID="7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"
Oct 30 22:12:59.271286 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:12:59.271265    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2\": container with ID starting with 7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2 not found: ID does not exist" containerID="7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"
Oct 30 22:12:59.271341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.271293    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2"} err="failed to get container status \"7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2\": rpc error: code = NotFound desc = could not find container \"7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2\": container with ID starting with 7ae4761893f09bd0476d6a11e9a0230824730a161936fae94292b6e620cfd7f2 not found: ID does not exist"
Oct 30 22:12:59.357450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.357411    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b41768-b63b-44df-94a1-292629eb7d29-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.357450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.357444    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pnj\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-kube-api-access-57pnj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.357450 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.357456    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b41768-b63b-44df-94a1-292629eb7d29-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.357838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.357465    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b41768-b63b-44df-94a1-292629eb7d29-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.423234 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.423206    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:12:59.559693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.559663    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5\" (UniqueName: \"kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5\") pod \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") "
Oct 30 22:12:59.559832 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.559774    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca\") pod \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") "
Oct 30 22:12:59.559900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.559879    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca\") pod \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\" (UID: \"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e\") "
Oct 30 22:12:59.560260 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.560236    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca" pod "c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" (UID: "c2da20a6-beb9-4e28-afb8-c6a04a7ed20e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:59.560492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.560474    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.562493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.562470    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca" pod "c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" (UID: "c2da20a6-beb9-4e28-afb8-c6a04a7ed20e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:59.562595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.562539    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5" pod "c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" (UID: "c2da20a6-beb9-4e28-afb8-c6a04a7ed20e"). InnerVolumeSpecName "kube-api-access-s9dv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:59.596702 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.596668    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"]
Oct 30 22:12:59.601172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.601140    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c19461-openshift-default-598555bd89-fjwkm"]
Oct 30 22:12:59.661820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.661794    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dv5\" (UniqueName: \"kubernetes.io/projected/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-kube-api-access-s9dv5\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.661820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.661821    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:12:59.990594 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:12:59.990567    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 22:13:00.031971 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.031914    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 22:13:00.067024 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.066988    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c\" (UniqueName: \"kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c\") pod \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") "
Oct 30 22:13:00.067269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.067252    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume\") pod \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\" (UID: \"20f65d1a-3a47-44ce-9a6b-37128be4a3f0\") "
Oct 30 22:13:00.067637 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.067596    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume" pod "20f65d1a-3a47-44ce-9a6b-37128be4a3f0" (UID: "20f65d1a-3a47-44ce-9a6b-37128be4a3f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:00.067824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.067810    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:00.069736 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.069712    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c" pod "20f65d1a-3a47-44ce-9a6b-37128be4a3f0" (UID: "20f65d1a-3a47-44ce-9a6b-37128be4a3f0"). InnerVolumeSpecName "kube-api-access-csr7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:00.169112 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.169078    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9\" (UniqueName: \"kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9\") pod \"bf18df61-47dc-493e-957c-7c004dca33d8\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") "
Oct 30 22:13:00.169302 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.169206    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume\") pod \"bf18df61-47dc-493e-957c-7c004dca33d8\" (UID: \"bf18df61-47dc-493e-957c-7c004dca33d8\") "
Oct 30 22:13:00.169642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.169594    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume" pod "bf18df61-47dc-493e-957c-7c004dca33d8" (UID: "bf18df61-47dc-493e-957c-7c004dca33d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:00.169869 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.169849    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csr7c\" (UniqueName: \"kubernetes.io/projected/20f65d1a-3a47-44ce-9a6b-37128be4a3f0-kube-api-access-csr7c\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:00.169965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.169871    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf18df61-47dc-493e-957c-7c004dca33d8-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:00.171383 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.171364    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9" pod "bf18df61-47dc-493e-957c-7c004dca33d8" (UID: "bf18df61-47dc-493e-957c-7c004dca33d8"). InnerVolumeSpecName "kube-api-access-l9pl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:00.259072 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.259041    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"
Oct 30 22:13:00.259475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.259070    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h" event={"ID":"c2da20a6-beb9-4e28-afb8-c6a04a7ed20e","Type":"ContainerDied","Data":"85e6f0b306e9428402f4f36c9e511368ed59ce6c3dabe71939037f63f56f5180"}
Oct 30 22:13:00.259475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.259119    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e6f0b306e9428402f4f36c9e511368ed59ce6c3dabe71939037f63f56f5180"
Oct 30 22:13:00.260918 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.260894    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c" event={"ID":"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a","Type":"ContainerStarted","Data":"6233fbee46feac948258d1158af328b490f4617af271b127ea6d473be2ba2c77"}
Oct 30 22:13:00.262315 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.262287    6695 generic.go:356] "Generic (PLEG): container finished" podID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerID="f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed" exitCode=0
Oct 30 22:13:00.262419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.262320    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"
Oct 30 22:13:00.262419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.262361    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" event={"ID":"20f65d1a-3a47-44ce-9a6b-37128be4a3f0","Type":"ContainerDied","Data":"f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed"}
Oct 30 22:13:00.262419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.262411    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5" event={"ID":"20f65d1a-3a47-44ce-9a6b-37128be4a3f0","Type":"ContainerDied","Data":"9b9c694429f9967becd773118af5967599a7f81adea9082683f539499fb7325a"}
Oct 30 22:13:00.262596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.262437    6695 scope.go:122] "RemoveContainer" containerID="f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed"
Oct 30 22:13:00.264396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.264373    6695 generic.go:356] "Generic (PLEG): container finished" podID="bf18df61-47dc-493e-957c-7c004dca33d8" containerID="cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a" exitCode=0
Oct 30 22:13:00.264481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.264419    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"
Oct 30 22:13:00.264481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.264448    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" event={"ID":"bf18df61-47dc-493e-957c-7c004dca33d8","Type":"ContainerDied","Data":"cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a"}
Oct 30 22:13:00.264640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.264496    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv" event={"ID":"bf18df61-47dc-493e-957c-7c004dca33d8","Type":"ContainerDied","Data":"a4cc90b17cd362132a5050549b6e99d5af5e67fea6be1bd82d1221e79bb29702"}
Oct 30 22:13:00.271157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.271129    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9pl9\" (UniqueName: \"kubernetes.io/projected/bf18df61-47dc-493e-957c-7c004dca33d8-kube-api-access-l9pl9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:00.300884 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.300864    6695 scope.go:122] "RemoveContainer" containerID="f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed"
Oct 30 22:13:00.301197 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:00.301175    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed\": container with ID starting with f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed not found: ID does not exist" containerID="f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed"
Oct 30 22:13:00.301272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.301208    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed"} err="failed to get container status \"f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed\": rpc error: code = NotFound desc = could not find container \"f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed\": container with ID starting with f2e4a110697e742b032095420bc6c21449d46796720597de4c2b7228375339ed not found: ID does not exist"
Oct 30 22:13:00.301272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.301233    6695 scope.go:122] "RemoveContainer" containerID="cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a"
Oct 30 22:13:00.312253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.312226    6695 scope.go:122] "RemoveContainer" containerID="cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a"
Oct 30 22:13:00.312573 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:00.312554    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a\": container with ID starting with cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a not found: ID does not exist" containerID="cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a"
Oct 30 22:13:00.312671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.312582    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a"} err="failed to get container status \"cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a\": rpc error: code = NotFound desc = could not find container \"cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a\": container with ID starting with cd89fe44f75821376891bbb2ce3554c4769c686c5796544869bbee9ff5f5bd6a not found: ID does not exist"
Oct 30 22:13:00.359769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.359732    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"]
Oct 30 22:13:00.364251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.364222    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19461-64c65d57dc-4khbv"]
Oct 30 22:13:00.397248 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.397212    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"]
Oct 30 22:13:00.403743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:00.403712    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19461-74fbf7595-rd5z5"]
Oct 30 22:13:01.074948 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:01.074910    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:13:01.215671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.215630    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" path="/var/lib/kubelet/pods/20f65d1a-3a47-44ce-9a6b-37128be4a3f0/volumes"
Oct 30 22:13:01.216089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.216073    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" path="/var/lib/kubelet/pods/bf18df61-47dc-493e-957c-7c004dca33d8/volumes"
Oct 30 22:13:01.216512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.216497    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" path="/var/lib/kubelet/pods/d0b41768-b63b-44df-94a1-292629eb7d29/volumes"
Oct 30 22:13:01.276164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.276128    6695 generic.go:356] "Generic (PLEG): container finished" podID="f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" containerID="6233fbee46feac948258d1158af328b490f4617af271b127ea6d473be2ba2c77" exitCode=0
Oct 30 22:13:01.276663 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.276221    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c" event={"ID":"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a","Type":"ContainerDied","Data":"6233fbee46feac948258d1158af328b490f4617af271b127ea6d473be2ba2c77"}
Oct 30 22:13:01.438176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.438144    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:13:01.486855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.486819    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca\") pod \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") "
Oct 30 22:13:01.486855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.486857    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca\") pod \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") "
Oct 30 22:13:01.487125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.486984    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86\" (UniqueName: \"kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86\") pod \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\" (UID: \"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a\") "
Oct 30 22:13:01.487267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.487232    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca" pod "f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" (UID: "f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:01.487439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.487383    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:01.489045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.489016    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca" pod "f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" (UID: "f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:01.489153 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.489124    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86" pod "f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" (UID: "f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a"). InnerVolumeSpecName "kube-api-access-tdl86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:01.588362 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.588324    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:01.588362 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.588354    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdl86\" (UniqueName: \"kubernetes.io/projected/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a-kube-api-access-tdl86\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:01.827771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:01.827733    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:02.285543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:02.285515    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"
Oct 30 22:13:02.285965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:02.285547    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c" event={"ID":"f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a","Type":"ContainerDied","Data":"7b486676c9e7e7219d1c2834fdbcad08166671a345cd229bd7a55ae2c135273c"}
Oct 30 22:13:02.285965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:02.285596    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b486676c9e7e7219d1c2834fdbcad08166671a345cd229bd7a55ae2c135273c"
Oct 30 22:13:03.097564 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:03.097528    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2f7991_55c5_4109_b142_d8371f73f09c.slice/crio-conmon-7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2f7991_55c5_4109_b142_d8371f73f09c.slice/crio-7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:13:03.298104 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.298030    6695 generic.go:356] "Generic (PLEG): container finished" podID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerID="7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550" exitCode=0
Oct 30 22:13:03.298533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.298100    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" event={"ID":"3c2f7991-55c5-4109-b142-d8371f73f09c","Type":"ContainerDied","Data":"7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550"}
Oct 30 22:13:03.298533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.298175    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn" event={"ID":"3c2f7991-55c5-4109-b142-d8371f73f09c","Type":"ContainerDied","Data":"72f0ccaaa0e2724f190f25c4b7884a6a99155ca948e03035eb5a818b5f34a05c"}
Oct 30 22:13:03.298533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.298203    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f0ccaaa0e2724f190f25c4b7884a6a99155ca948e03035eb5a818b5f34a05c"
Oct 30 22:13:03.298533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.298223    6695 scope.go:122] "RemoveContainer" containerID="2e32d1804e8470755d00c3996eb58f19cbe8e8e57a4de5808503c5974fb8e3f5"
Oct 30 22:13:03.303533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.303509    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:13:03.414401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414366    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414438    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414508    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414540    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414575    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414646    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414694    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414741    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.414845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414731    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:03.414845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414795    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:03.414845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414823    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data\") pod \"3c2f7991-55c5-4109-b142-d8371f73f09c\" (UID: \"3c2f7991-55c5-4109-b142-d8371f73f09c\") "
Oct 30 22:13:03.415196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.414939    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:03.415196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415047    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:03.415312 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415274    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:03.415511 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415492    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.415645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415512    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.415645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415524    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.415645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415533    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2f7991-55c5-4109-b142-d8371f73f09c-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.415645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.415540    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.416997 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.416968    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:03.417134 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.417108    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "kube-api-access-trfpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:03.417346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.417324    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:03.417404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.417357    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy" pod "3c2f7991-55c5-4109-b142-d8371f73f09c" (UID: "3c2f7991-55c5-4109-b142-d8371f73f09c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:03.516958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.516923    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.516958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.516953    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfpl\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-kube-api-access-trfpl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.516958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.516963    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:03.517211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:03.516972    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2f7991-55c5-4109-b142-d8371f73f09c-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:04.307689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:04.307654    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"
Oct 30 22:13:04.401180 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:04.401147    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"]
Oct 30 22:13:04.404123 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:04.404097    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5454af-openshift-default-64cf7c5ddf-9nnvn"]
Oct 30 22:13:05.215652 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:05.215591    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" path="/var/lib/kubelet/pods/3c2f7991-55c5-4109-b142-d8371f73f09c/volumes"
Oct 30 22:13:07.012385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:07.012347    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"]
Oct 30 22:13:08.714591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.714530    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:13:08.800050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.800016    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume\") pod \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") "
Oct 30 22:13:08.800050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.800067    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp\" (UniqueName: \"kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp\") pod \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\" (UID: \"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d\") "
Oct 30 22:13:08.800532 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.800500    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume" pod "f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" (UID: "f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:08.800798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.800774    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:08.802372 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.802347    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp" pod "f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" (UID: "f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d"). InnerVolumeSpecName "kube-api-access-5q8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:08.901653 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:08.901592    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q8bp\" (UniqueName: \"kubernetes.io/projected/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d-kube-api-access-5q8bp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:09.026499 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.026468    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"]
Oct 30 22:13:09.058248 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.058208    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"]
Oct 30 22:13:09.346150 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.346052    6695 generic.go:356] "Generic (PLEG): container finished" podID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerID="02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8" exitCode=0
Oct 30 22:13:09.346150 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.346096    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"
Oct 30 22:13:09.346424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.346138    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" event={"ID":"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d","Type":"ContainerDied","Data":"02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8"}
Oct 30 22:13:09.346424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.346198    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl" event={"ID":"f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d","Type":"ContainerDied","Data":"6f8c7242cbe75934314ab8c9a873a923eae097fc143e797fdeb01a9afc8cf036"}
Oct 30 22:13:09.346424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.346227    6695 scope.go:122] "RemoveContainer" containerID="02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8"
Oct 30 22:13:09.356700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.356681    6695 scope.go:122] "RemoveContainer" containerID="02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8"
Oct 30 22:13:09.356992 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:09.356966    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8\": container with ID starting with 02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8 not found: ID does not exist" containerID="02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8"
Oct 30 22:13:09.357041 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.356999    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8"} err="failed to get container status \"02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8\": rpc error: code = NotFound desc = could not find container \"02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8\": container with ID starting with 02836ee1e43327533638818d0f0e4900156b7a56467c2f2282597106baf8cbc8 not found: ID does not exist"
Oct 30 22:13:09.403125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.403089    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"]
Oct 30 22:13:09.406708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:09.406673    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a42c31-7f56cb75f7-z4rbl"]
Oct 30 22:13:11.214479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:11.214447    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" path="/var/lib/kubelet/pods/f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d/volumes"
Oct 30 22:13:12.013758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:12.013708    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" podUID="47f91004-2572-4630-a24d-316cf63d8594" containerName="payload-pre-processing" containerID="cri-o://d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085" gracePeriod=25
Oct 30 22:13:14.027166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.027129    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht" podUID="58568bf1-662b-4a1c-aac6-e43ea30616cf" containerName="payload-processing" containerID="cri-o://453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1" gracePeriod=25
Oct 30 22:13:14.059464 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.059412    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerName="payload-pre-processing" containerID="cri-o://b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0" gracePeriod=25
Oct 30 22:13:14.220154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.220119    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" podUID="47f91004-2572-4630-a24d-316cf63d8594" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:14.294027 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.293968    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:13:14.367305 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.367260    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn\" (UniqueName: \"kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn\") pod \"58568bf1-662b-4a1c-aac6-e43ea30616cf\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") "
Oct 30 22:13:14.367523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.367337    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume\") pod \"58568bf1-662b-4a1c-aac6-e43ea30616cf\" (UID: \"58568bf1-662b-4a1c-aac6-e43ea30616cf\") "
Oct 30 22:13:14.368058 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.368028    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume" pod "58568bf1-662b-4a1c-aac6-e43ea30616cf" (UID: "58568bf1-662b-4a1c-aac6-e43ea30616cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:14.369686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.369657    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn" pod "58568bf1-662b-4a1c-aac6-e43ea30616cf" (UID: "58568bf1-662b-4a1c-aac6-e43ea30616cf"). InnerVolumeSpecName "kube-api-access-whlbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:14.387274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.387246    6695 generic.go:356] "Generic (PLEG): container finished" podID="58568bf1-662b-4a1c-aac6-e43ea30616cf" containerID="453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1" exitCode=0
Oct 30 22:13:14.387389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.387304    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"
Oct 30 22:13:14.387389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.387324    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht" event={"ID":"58568bf1-662b-4a1c-aac6-e43ea30616cf","Type":"ContainerDied","Data":"453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1"}
Oct 30 22:13:14.387389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.387386    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht" event={"ID":"58568bf1-662b-4a1c-aac6-e43ea30616cf","Type":"ContainerDied","Data":"f74b664e1eb45a15a587dee56602dbffa97fbbee97d66d9701001c485fc35b46"}
Oct 30 22:13:14.387519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.387410    6695 scope.go:122] "RemoveContainer" containerID="453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1"
Oct 30 22:13:14.404582 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.404462    6695 scope.go:122] "RemoveContainer" containerID="453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1"
Oct 30 22:13:14.404902 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:14.404872    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1\": container with ID starting with 453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1 not found: ID does not exist" containerID="453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1"
Oct 30 22:13:14.405013 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.404915    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1"} err="failed to get container status \"453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1\": rpc error: code = NotFound desc = could not find container \"453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1\": container with ID starting with 453ff78a4397ffc0c5fd970c287639292350f52f8aebe691f12ac105f32578e1 not found: ID does not exist"
Oct 30 22:13:14.443990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.443954    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"]
Oct 30 22:13:14.447210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.447186    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-589ebc-5f6cdcbd47-fq4ht"]
Oct 30 22:13:14.468735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.468703    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whlbn\" (UniqueName: \"kubernetes.io/projected/58568bf1-662b-4a1c-aac6-e43ea30616cf-kube-api-access-whlbn\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:14.468735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:14.468732    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58568bf1-662b-4a1c-aac6-e43ea30616cf-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:15.214904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:15.214872    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58568bf1-662b-4a1c-aac6-e43ea30616cf" path="/var/lib/kubelet/pods/58568bf1-662b-4a1c-aac6-e43ea30616cf/volumes"
Oct 30 22:13:16.791416 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:16.791375    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:21.522427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:21.522382    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"]
Oct 30 22:13:21.533400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:21.533368    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-846656-bn4fp"]
Oct 30 22:13:21.903209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:21.903117    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"]
Oct 30 22:13:21.915755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:21.915717    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-589ebc-fv55c"]
Oct 30 22:13:22.340430 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:22.340388    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"]
Oct 30 22:13:22.351327 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:22.351290    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19a725-8ft9h"]
Oct 30 22:13:23.215625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:23.215574    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" path="/var/lib/kubelet/pods/c2da20a6-beb9-4e28-afb8-c6a04a7ed20e/volumes"
Oct 30 22:13:23.216021 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:23.216009    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e832671e-ed97-4170-9997-26bcda01fec3" path="/var/lib/kubelet/pods/e832671e-ed97-4170-9997-26bcda01fec3/volumes"
Oct 30 22:13:23.216402 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:23.216389    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" path="/var/lib/kubelet/pods/f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a/volumes"
Oct 30 22:13:24.221005 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:24.220970    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" podUID="47f91004-2572-4630-a24d-316cf63d8594" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:26.791881 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:26.791834    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:29.455657 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:29.455554    6695 scope.go:122] "RemoveContainer" containerID="7a72de2ab3c90b6cc94fd0057ba30866667309a08358d85befeb534bace5e550"
Oct 30 22:13:29.466192 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:29.466171    6695 scope.go:122] "RemoveContainer" containerID="e7db03b5eb13321d2975686c7acdd0f1a9426fb3112bafe1a873e70056df7360"
Oct 30 22:13:29.475830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:29.475807    6695 scope.go:122] "RemoveContainer" containerID="de7070be8f8d80cd73db8b00d0dc6c76b0f7325d2a1f8f5dacfd7a521c3558f2"
Oct 30 22:13:29.489124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:29.489096    6695 scope.go:122] "RemoveContainer" containerID="dc7dd0c2beef819152698c115604c8bdaa84ed5ce9b03f46010ac75000fd9354"
Oct 30 22:13:29.499502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:29.499478    6695 scope.go:122] "RemoveContainer" containerID="84c760b8cc91598e017011ebdb9762a157898ff95f309e28cecd7c878f15b652"
Oct 30 22:13:31.074552 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:31.074507    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:13:31.092671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.092641    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"]
Oct 30 22:13:31.092882 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.092851    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy" containerID="cri-o://80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129" gracePeriod=30
Oct 30 22:13:31.347502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.347427    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"]
Oct 30 22:13:31.347671 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.347646    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy" containerID="cri-o://75df84fe3c12157a9963e417637e44c2aac700b5dbe67147f8f7dca61f6d83b9" gracePeriod=30
Oct 30 22:13:31.690452 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.690375    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"]
Oct 30 22:13:31.691003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.690988    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy"
Oct 30 22:13:31.691061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691006    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy"
Oct 30 22:13:31.691061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691014    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" containerName="revoke-keys"
Oct 30 22:13:31.691061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691020    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" containerName="revoke-keys"
Oct 30 22:13:31.691061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691047    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerName="payload-processing"
Oct 30 22:13:31.691061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691052    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerName="payload-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691065    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691070    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691081    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" containerName="revoke-keys"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691085    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" containerName="revoke-keys"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691100    6695 cpu_manager.go:414] "containerMap: removing container" podUID="58568bf1-662b-4a1c-aac6-e43ea30616cf" containerName="payload-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691105    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58568bf1-662b-4a1c-aac6-e43ea30616cf" containerName="payload-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691121    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" containerName="payload-pre-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691127    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" containerName="payload-pre-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691139    6695 cpu_manager.go:414] "containerMap: removing container" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerName="payload-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691145    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerName="payload-processing"
Oct 30 22:13:31.691230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691228    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89a2d5d-4cbd-41b0-bd87-e5a50e9b1e7d" containerName="payload-processing"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691238    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691244    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2da20a6-beb9-4e28-afb8-c6a04a7ed20e" containerName="revoke-keys"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691252    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="58568bf1-662b-4a1c-aac6-e43ea30616cf" containerName="payload-processing"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691260    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf18df61-47dc-493e-957c-7c004dca33d8" containerName="payload-pre-processing"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691266    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f65d1a-3a47-44ce-9a6b-37128be4a3f0" containerName="payload-processing"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691273    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691279    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28ee9c0-ef7e-47c3-b9b6-e44a4f427f3a" containerName="revoke-keys"
Oct 30 22:13:31.691595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.691286    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy"
Oct 30 22:13:31.697048 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.697028    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.718547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.718526    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"]
Oct 30 22:13:31.720789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.720770    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:31.720987 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.720967    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:31.721110 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.721094    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:13:31.721184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.721094    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:31.824132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.824095    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.824345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.824231    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.824345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.824327    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4xd\" (UniqueName: \"kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.896106 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.896070    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"]
Oct 30 22:13:31.896755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.896738    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy"
Oct 30 22:13:31.896755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.896755    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2f7991-55c5-4109-b142-d8371f73f09c" containerName="istio-proxy"
Oct 30 22:13:31.896911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.896788    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy"
Oct 30 22:13:31.896911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.896793    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy"
Oct 30 22:13:31.896911 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.896909    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b41768-b63b-44df-94a1-292629eb7d29" containerName="istio-proxy"
Oct 30 22:13:31.900567 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.900547    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:31.920946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.920912    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"]
Oct 30 22:13:31.925723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.925689    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4xd\" (UniqueName: \"kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.925929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.925909    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.926068 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.926051    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.926672 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.926645    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.929210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.929183    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:31.953167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:31.953091    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4xd\" (UniqueName: \"kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:32.010493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.010460    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:32.027423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.027392    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.027529 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.027458    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wqg\" (UniqueName: \"kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.027707 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.027684    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.129761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.129723    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.130229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.129812    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.130229 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.129868    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wqg\" (UniqueName: \"kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.130933 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.130910    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.132243 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.132218    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.161046 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.161016    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wqg\" (UniqueName: \"kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg\") pod \"maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.216030 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.215958    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:32.262732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.262701    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:13:32.327352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.327318    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"]
Oct 30 22:13:32.434291 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.434258    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm\" (UniqueName: \"kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm\") pod \"47f91004-2572-4630-a24d-316cf63d8594\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") "
Oct 30 22:13:32.434475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.434304    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume\") pod \"47f91004-2572-4630-a24d-316cf63d8594\" (UID: \"47f91004-2572-4630-a24d-316cf63d8594\") "
Oct 30 22:13:32.434743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.434711    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume" pod "47f91004-2572-4630-a24d-316cf63d8594" (UID: "47f91004-2572-4630-a24d-316cf63d8594"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:32.436529 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.436508    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm" pod "47f91004-2572-4630-a24d-316cf63d8594" (UID: "47f91004-2572-4630-a24d-316cf63d8594"). InnerVolumeSpecName "kube-api-access-x6rpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:32.538958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.538909    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rpm\" (UniqueName: \"kubernetes.io/projected/47f91004-2572-4630-a24d-316cf63d8594-kube-api-access-x6rpm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:32.538958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.538946    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f91004-2572-4630-a24d-316cf63d8594-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:32.546343 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.545902    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7" event={"ID":"31334a4f-e7ba-47bc-b0d6-ce0c98acea82","Type":"ContainerStarted","Data":"6ae0f58a009f0ee6033edaed9f5c2a89d2e5d48509b5d008d5421012e80a512e"}
Oct 30 22:13:32.551630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.548261    6695 generic.go:356] "Generic (PLEG): container finished" podID="47f91004-2572-4630-a24d-316cf63d8594" containerID="d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085" exitCode=0
Oct 30 22:13:32.551630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.548521    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" event={"ID":"47f91004-2572-4630-a24d-316cf63d8594","Type":"ContainerDied","Data":"d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085"}
Oct 30 22:13:32.551630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.548571    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j" event={"ID":"47f91004-2572-4630-a24d-316cf63d8594","Type":"ContainerDied","Data":"d0d529fb17da9f4e820f8f65218d5228680eab6370b10eaf3a9fa401558e28fa"}
Oct 30 22:13:32.551630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.548617    6695 scope.go:122] "RemoveContainer" containerID="d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085"
Oct 30 22:13:32.551630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.548775    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"
Oct 30 22:13:32.551630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.549395    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"]
Oct 30 22:13:32.581575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.581495    6695 scope.go:122] "RemoveContainer" containerID="d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085"
Oct 30 22:13:32.581838 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:32.581811    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085\": container with ID starting with d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085 not found: ID does not exist" containerID="d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085"
Oct 30 22:13:32.581942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.581846    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085"} err="failed to get container status \"d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085\": rpc error: code = NotFound desc = could not find container \"d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085\": container with ID starting with d082bf6eb7acb6f616f4a501449442611a5278811a82e6bd5062b1d9682bf085 not found: ID does not exist"
Oct 30 22:13:32.609410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.609388    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"]
Oct 30 22:13:32.613688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.613664    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19a725-65dd74cd5f-nsd4j"]
Oct 30 22:13:32.949806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.949772    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"]
Oct 30 22:13:32.950755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.950735    6695 cpu_manager.go:414] "containerMap: removing container" podUID="47f91004-2572-4630-a24d-316cf63d8594" containerName="payload-pre-processing"
Oct 30 22:13:32.950823 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.950757    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f91004-2572-4630-a24d-316cf63d8594" containerName="payload-pre-processing"
Oct 30 22:13:32.950894 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.950885    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f91004-2572-4630-a24d-316cf63d8594" containerName="payload-pre-processing"
Oct 30 22:13:32.954448 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.954428    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:32.979090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:32.979058    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"]
Oct 30 22:13:33.045686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.045644    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.045876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.045707    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.045876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.045837    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8drc\" (UniqueName: \"kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.147492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.147397    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.147492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.147460    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.147981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.147511    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8drc\" (UniqueName: \"kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.148149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.148126    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.149900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.149880    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.170276 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.170250    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8drc\" (UniqueName: \"kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc\") pod \"maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.221639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.221584    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f91004-2572-4630-a24d-316cf63d8594" path="/var/lib/kubelet/pods/47f91004-2572-4630-a24d-316cf63d8594/volumes"
Oct 30 22:13:33.267829 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.267796    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:33.557559 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.557525    6695 generic.go:356] "Generic (PLEG): container finished" podID="31334a4f-e7ba-47bc-b0d6-ce0c98acea82" containerID="3465ec156babd347cec00314124fbc61370f4ac74216876a5f8e28c93feaeac6" exitCode=0
Oct 30 22:13:33.557763 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.557628    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7" event={"ID":"31334a4f-e7ba-47bc-b0d6-ce0c98acea82","Type":"ContainerDied","Data":"3465ec156babd347cec00314124fbc61370f4ac74216876a5f8e28c93feaeac6"}
Oct 30 22:13:33.560473 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.560446    6695 generic.go:356] "Generic (PLEG): container finished" podID="c8bec463-2506-438a-bc34-c177ced34fea" containerID="45ae2aa2bd78d253d2525de98b9fec1a0d694d7c320e75182b1cb3b818a41861" exitCode=0
Oct 30 22:13:33.560641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.560474    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq" event={"ID":"c8bec463-2506-438a-bc34-c177ced34fea","Type":"ContainerDied","Data":"45ae2aa2bd78d253d2525de98b9fec1a0d694d7c320e75182b1cb3b818a41861"}
Oct 30 22:13:33.560641 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.560520    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq" event={"ID":"c8bec463-2506-438a-bc34-c177ced34fea","Type":"ContainerStarted","Data":"8bc0bcdcb898fc66de25f1dc6fe5a1081ad06b812744fda85f3c133ed37c03df"}
Oct 30 22:13:33.579328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:33.579306    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"]
Oct 30 22:13:34.332597 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.332567    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:13:34.465172 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.465146    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume\") pod \"69963d5e-752a-4fe1-835e-23f0a83a58a3\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") "
Oct 30 22:13:34.465288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.465239    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx\" (UniqueName: \"kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx\") pod \"69963d5e-752a-4fe1-835e-23f0a83a58a3\" (UID: \"69963d5e-752a-4fe1-835e-23f0a83a58a3\") "
Oct 30 22:13:34.465519 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.465493    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume" pod "69963d5e-752a-4fe1-835e-23f0a83a58a3" (UID: "69963d5e-752a-4fe1-835e-23f0a83a58a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:34.465778 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.465758    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69963d5e-752a-4fe1-835e-23f0a83a58a3-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.467377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.467355    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx" pod "69963d5e-752a-4fe1-835e-23f0a83a58a3" (UID: "69963d5e-752a-4fe1-835e-23f0a83a58a3"). InnerVolumeSpecName "kube-api-access-4hhdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:34.567307 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.567277    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhdx\" (UniqueName: \"kubernetes.io/projected/69963d5e-752a-4fe1-835e-23f0a83a58a3-kube-api-access-4hhdx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.570339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.570307    6695 generic.go:356] "Generic (PLEG): container finished" podID="77679b84-bbb2-4efe-b913-9a9b676f7ac3" containerID="dd016c309d7935e1e62b176c3428b341e70d511c3ec61695be9c9eab55d53fd1" exitCode=0
Oct 30 22:13:34.570500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.570388    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8" event={"ID":"77679b84-bbb2-4efe-b913-9a9b676f7ac3","Type":"ContainerDied","Data":"dd016c309d7935e1e62b176c3428b341e70d511c3ec61695be9c9eab55d53fd1"}
Oct 30 22:13:34.570500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.570445    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8" event={"ID":"77679b84-bbb2-4efe-b913-9a9b676f7ac3","Type":"ContainerStarted","Data":"98866ce93d385ec49752a24ef0786fa0340996fe7371246aa1dd3a17313afc3e"}
Oct 30 22:13:34.572113 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.572087    6695 generic.go:356] "Generic (PLEG): container finished" podID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerID="b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0" exitCode=0
Oct 30 22:13:34.572236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.572124    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"
Oct 30 22:13:34.572236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.572161    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" event={"ID":"69963d5e-752a-4fe1-835e-23f0a83a58a3","Type":"ContainerDied","Data":"b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0"}
Oct 30 22:13:34.572236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.572226    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk" event={"ID":"69963d5e-752a-4fe1-835e-23f0a83a58a3","Type":"ContainerDied","Data":"1af720e4f52835769d61b77c6873ff24954072f52c07fd926e537e802094b310"}
Oct 30 22:13:34.572373 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.572263    6695 scope.go:122] "RemoveContainer" containerID="b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0"
Oct 30 22:13:34.584766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.584744    6695 scope.go:122] "RemoveContainer" containerID="b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0"
Oct 30 22:13:34.585116 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:34.585092    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0\": container with ID starting with b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0 not found: ID does not exist" containerID="b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0"
Oct 30 22:13:34.585227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.585127    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0"} err="failed to get container status \"b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0\": rpc error: code = NotFound desc = could not find container \"b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0\": container with ID starting with b34c398e150735bad4059d1e7c657bc972efff37f3010de1acdcae048bebdfd0 not found: ID does not exist"
Oct 30 22:13:34.666796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.666761    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"]
Oct 30 22:13:34.683106 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.683074    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-589ebc-5c658d849f-g4dxk"]
Oct 30 22:13:34.712784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.712722    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:34.749733 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.749709    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:34.870994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.870964    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd\" (UniqueName: \"kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd\") pod \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") "
Oct 30 22:13:34.871190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871008    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca\") pod \"c8bec463-2506-438a-bc34-c177ced34fea\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") "
Oct 30 22:13:34.871190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871056    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca\") pod \"c8bec463-2506-438a-bc34-c177ced34fea\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") "
Oct 30 22:13:34.871330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871204    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca\") pod \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") "
Oct 30 22:13:34.871330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871250    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca\") pod \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\" (UID: \"31334a4f-e7ba-47bc-b0d6-ce0c98acea82\") "
Oct 30 22:13:34.871439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871397    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg\" (UniqueName: \"kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg\") pod \"c8bec463-2506-438a-bc34-c177ced34fea\" (UID: \"c8bec463-2506-438a-bc34-c177ced34fea\") "
Oct 30 22:13:34.871550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871520    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca" pod "c8bec463-2506-438a-bc34-c177ced34fea" (UID: "c8bec463-2506-438a-bc34-c177ced34fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:34.871761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.871644    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca" pod "31334a4f-e7ba-47bc-b0d6-ce0c98acea82" (UID: "31334a4f-e7ba-47bc-b0d6-ce0c98acea82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:34.872206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.872179    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.872319 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.872212    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.873471 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.873442    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca" pod "c8bec463-2506-438a-bc34-c177ced34fea" (UID: "c8bec463-2506-438a-bc34-c177ced34fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:34.873584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.873565    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd" pod "31334a4f-e7ba-47bc-b0d6-ce0c98acea82" (UID: "31334a4f-e7ba-47bc-b0d6-ce0c98acea82"). InnerVolumeSpecName "kube-api-access-lk4xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:34.873720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.873597    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca" pod "31334a4f-e7ba-47bc-b0d6-ce0c98acea82" (UID: "31334a4f-e7ba-47bc-b0d6-ce0c98acea82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:34.873720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.873689    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg" pod "c8bec463-2506-438a-bc34-c177ced34fea" (UID: "c8bec463-2506-438a-bc34-c177ced34fea"). InnerVolumeSpecName "kube-api-access-79wqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:34.973664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.973579    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wqg\" (UniqueName: \"kubernetes.io/projected/c8bec463-2506-438a-bc34-c177ced34fea-kube-api-access-79wqg\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.973664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.973640    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4xd\" (UniqueName: \"kubernetes.io/projected/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-kube-api-access-lk4xd\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.973664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.973653    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8bec463-2506-438a-bc34-c177ced34fea-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:34.973664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:34.973663    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31334a4f-e7ba-47bc-b0d6-ce0c98acea82-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:35.215305 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.215266    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" path="/var/lib/kubelet/pods/69963d5e-752a-4fe1-835e-23f0a83a58a3/volumes"
Oct 30 22:13:35.583071 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.583038    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"
Oct 30 22:13:35.583500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.583074    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7" event={"ID":"31334a4f-e7ba-47bc-b0d6-ce0c98acea82","Type":"ContainerDied","Data":"6ae0f58a009f0ee6033edaed9f5c2a89d2e5d48509b5d008d5421012e80a512e"}
Oct 30 22:13:35.583500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.583124    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae0f58a009f0ee6033edaed9f5c2a89d2e5d48509b5d008d5421012e80a512e"
Oct 30 22:13:35.585915 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.585889    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq" event={"ID":"c8bec463-2506-438a-bc34-c177ced34fea","Type":"ContainerDied","Data":"8bc0bcdcb898fc66de25f1dc6fe5a1081ad06b812744fda85f3c133ed37c03df"}
Oct 30 22:13:35.586033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.585934    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bc0bcdcb898fc66de25f1dc6fe5a1081ad06b812744fda85f3c133ed37c03df"
Oct 30 22:13:35.586033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.585960    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"
Oct 30 22:13:35.738472 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.738445    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:35.783134 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.783082    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca\") pod \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") "
Oct 30 22:13:35.783438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.783412    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca\") pod \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") "
Oct 30 22:13:35.783704 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.783685    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc\" (UniqueName: \"kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc\") pod \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\" (UID: \"77679b84-bbb2-4efe-b913-9a9b676f7ac3\") "
Oct 30 22:13:35.783895 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.783873    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca" pod "77679b84-bbb2-4efe-b913-9a9b676f7ac3" (UID: "77679b84-bbb2-4efe-b913-9a9b676f7ac3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:35.784645 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.784593    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:35.785690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.785661    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca" pod "77679b84-bbb2-4efe-b913-9a9b676f7ac3" (UID: "77679b84-bbb2-4efe-b913-9a9b676f7ac3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:35.786086 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.786061    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc" pod "77679b84-bbb2-4efe-b913-9a9b676f7ac3" (UID: "77679b84-bbb2-4efe-b913-9a9b676f7ac3"). InnerVolumeSpecName "kube-api-access-d8drc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:35.885544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.885456    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77679b84-bbb2-4efe-b913-9a9b676f7ac3-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:35.885544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:35.885491    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8drc\" (UniqueName: \"kubernetes.io/projected/77679b84-bbb2-4efe-b913-9a9b676f7ac3-kube-api-access-d8drc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.427083 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.427050    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 22:13:36.499264 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.499220    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.499494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.499417    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.499576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.499497    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.500059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.500026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.500175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.500077    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.500175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.500115    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.500293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.500191    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.500293 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.500222    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.500427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.500349    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs\") pod \"8c656625-0f5d-422f-a718-c447c368fcf6\" (UID: \"8c656625-0f5d-422f-a718-c447c368fcf6\") "
Oct 30 22:13:36.501100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.501068    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:36.501374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.501166    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.501374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.501222    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.501530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.501322    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.501530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.501352    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.502331 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.502302    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:36.502872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.502798    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:36.510404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.510377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.511578 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.511557    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6" pod "8c656625-0f5d-422f-a718-c447c368fcf6" (UID: "8c656625-0f5d-422f-a718-c447c368fcf6"). InnerVolumeSpecName "kube-api-access-7lpj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:36.597753 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.597718    6695 generic.go:356] "Generic (PLEG): container finished" podID="fe820873-6dc6-44a7-8813-6902c183c856" containerID="75df84fe3c12157a9963e417637e44c2aac700b5dbe67147f8f7dca61f6d83b9" exitCode=0
Oct 30 22:13:36.598214 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.597784    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" event={"ID":"fe820873-6dc6-44a7-8813-6902c183c856","Type":"ContainerDied","Data":"75df84fe3c12157a9963e417637e44c2aac700b5dbe67147f8f7dca61f6d83b9"}
Oct 30 22:13:36.598214 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.597849    6695 scope.go:122] "RemoveContainer" containerID="304cf1235d6d2691f6e379c9f4b81120e0fcb7c6aa78b0487bcea5f42a910712"
Oct 30 22:13:36.600069 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.600037    6695 generic.go:356] "Generic (PLEG): container finished" podID="8c656625-0f5d-422f-a718-c447c368fcf6" containerID="80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129" exitCode=0
Oct 30 22:13:36.600176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.600093    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"
Oct 30 22:13:36.600176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.600111    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" event={"ID":"8c656625-0f5d-422f-a718-c447c368fcf6","Type":"ContainerDied","Data":"80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129"}
Oct 30 22:13:36.600176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.600162    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h" event={"ID":"8c656625-0f5d-422f-a718-c447c368fcf6","Type":"ContainerDied","Data":"4d952e255e9c0c4d206d88cd1abd9a45a9d019686cff98cd1c9cd9ba8bb4dc17"}
Oct 30 22:13:36.603518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603494    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8" event={"ID":"77679b84-bbb2-4efe-b913-9a9b676f7ac3","Type":"ContainerDied","Data":"98866ce93d385ec49752a24ef0786fa0340996fe7371246aa1dd3a17313afc3e"}
Oct 30 22:13:36.603662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603524    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"
Oct 30 22:13:36.603662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603535    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98866ce93d385ec49752a24ef0786fa0340996fe7371246aa1dd3a17313afc3e"
Oct 30 22:13:36.603662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603577    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpj6\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-kube-api-access-7lpj6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.603662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603632    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c656625-0f5d-422f-a718-c447c368fcf6-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.603662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603651    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.603937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603767    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.603937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603908    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.603937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603923    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.603937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603935    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c656625-0f5d-422f-a718-c447c368fcf6-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.604147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603948    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c656625-0f5d-422f-a718-c447c368fcf6-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.604147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.603962    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c656625-0f5d-422f-a718-c447c368fcf6-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.615194 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.615175    6695 scope.go:122] "RemoveContainer" containerID="80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129"
Oct 30 22:13:36.626405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.626384    6695 scope.go:122] "RemoveContainer" containerID="4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"
Oct 30 22:13:36.626797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.626773    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:13:36.636178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.636157    6695 scope.go:122] "RemoveContainer" containerID="80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129"
Oct 30 22:13:36.636442 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:36.636420    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129\": container with ID starting with 80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129 not found: ID does not exist" containerID="80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129"
Oct 30 22:13:36.636509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.636451    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129"} err="failed to get container status \"80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129\": rpc error: code = NotFound desc = could not find container \"80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129\": container with ID starting with 80dee1533a035abd3a8bbdd1720acef7c52817ac38226bedc96a75e61bc13129 not found: ID does not exist"
Oct 30 22:13:36.636509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.636468    6695 scope.go:122] "RemoveContainer" containerID="4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"
Oct 30 22:13:36.636773 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:13:36.636756    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02\": container with ID starting with 4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02 not found: ID does not exist" containerID="4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"
Oct 30 22:13:36.636831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.636779    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02"} err="failed to get container status \"4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02\": rpc error: code = NotFound desc = could not find container \"4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02\": container with ID starting with 4ffb3b599c53378eca44ff5bf3f61a04e8d87c718cca544dd02f4ec851aecf02 not found: ID does not exist"
Oct 30 22:13:36.652627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.652589    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"]
Oct 30 22:13:36.658041 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.658011    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-lrn8h"]
Oct 30 22:13:36.704799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.704766    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.704988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.704834    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.704988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.704921    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705119 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705119 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705083    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705170    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705213    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705297    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.705381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705316    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705336    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:36.705381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705357    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data\") pod \"fe820873-6dc6-44a7-8813-6902c183c856\" (UID: \"fe820873-6dc6-44a7-8813-6902c183c856\") "
Oct 30 22:13:36.705628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705427    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.705689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705643    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.705689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705668    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.705973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705950    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.706054 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705979    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.706054 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.705995    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe820873-6dc6-44a7-8813-6902c183c856-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.706054 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.706008    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.706054 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.706023    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.707176 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.707156    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:36.707289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.707270    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:36.707432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.707415    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:36.707533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.707517    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8" pod "fe820873-6dc6-44a7-8813-6902c183c856" (UID: "fe820873-6dc6-44a7-8813-6902c183c856"). InnerVolumeSpecName "kube-api-access-r5km8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:36.807280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.807242    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5km8\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-kube-api-access-r5km8\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.807280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.807277    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe820873-6dc6-44a7-8813-6902c183c856-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.807280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.807288    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe820873-6dc6-44a7-8813-6902c183c856-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:36.807549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:36.807296    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe820873-6dc6-44a7-8813-6902c183c856-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.214157 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:37.214121    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" path="/var/lib/kubelet/pods/8c656625-0f5d-422f-a718-c447c368fcf6/volumes"
Oct 30 22:13:37.613306 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:37.613272    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"
Oct 30 22:13:37.613777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:37.613274    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx" event={"ID":"fe820873-6dc6-44a7-8813-6902c183c856","Type":"ContainerDied","Data":"1f10b6bf91a0cb04f9b799fdb5a12c383f8b1999f567306b5a76fa1f45146d3c"}
Oct 30 22:13:37.613777 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:37.613408    6695 scope.go:122] "RemoveContainer" containerID="75df84fe3c12157a9963e417637e44c2aac700b5dbe67147f8f7dca61f6d83b9"
Oct 30 22:13:37.666125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:37.666099    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"]
Oct 30 22:13:37.677244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:37.677217    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-589ebc-openshift-default-85454b884f-qk2zx"]
Oct 30 22:13:39.214496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:39.214466    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe820873-6dc6-44a7-8813-6902c183c856" path="/var/lib/kubelet/pods/fe820873-6dc6-44a7-8813-6902c183c856/volumes"
Oct 30 22:13:42.142214 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:42.142175    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"]
Oct 30 22:13:42.175731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:42.175690    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"]
Oct 30 22:13:42.931354 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:42.931319    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"]
Oct 30 22:13:42.956654 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:42.956626    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"]
Oct 30 22:13:43.742891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:43.742856    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"]
Oct 30 22:13:47.143640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:47.143575    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerName="payload-processing" containerID="cri-o://e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31" gracePeriod=25
Oct 30 22:13:47.176364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:47.176317    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerName="payload-pre-processing" containerID="cri-o://a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6" gracePeriod=25
Oct 30 22:13:47.931825 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:47.931779    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerName="payload-processing" containerID="cri-o://99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95" gracePeriod=25
Oct 30 22:13:47.957708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:47.957671    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerName="payload-pre-processing" containerID="cri-o://02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27" gracePeriod=25
Oct 30 22:13:48.744106 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:48.744054    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerName="payload-processing" containerID="cri-o://52de8f4e196b59d44947f5f309264cbe7c286df7a68bafff74ab9310395e9c4c" gracePeriod=25
Oct 30 22:13:49.014868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:49.014765    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"]
Oct 30 22:13:49.027029 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:49.027000    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-5d3601-xlvcq"]
Oct 30 22:13:49.290625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:49.283636    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bec463-2506-438a-bc34-c177ced34fea" path="/var/lib/kubelet/pods/c8bec463-2506-438a-bc34-c177ced34fea/volumes"
Oct 30 22:13:49.618889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:49.618791    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"]
Oct 30 22:13:49.642800 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:49.642762    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-805edc-wn2r8"]
Oct 30 22:13:51.213916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:51.213882    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77679b84-bbb2-4efe-b913-9a9b676f7ac3" path="/var/lib/kubelet/pods/77679b84-bbb2-4efe-b913-9a9b676f7ac3/volumes"
Oct 30 22:13:51.571636 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:51.571573    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:51.801936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:51.801900    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:52.517300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:52.517248    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:52.517300 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:52.517283    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:52.875277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:52.875237    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:56.926863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:56.926822    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"]
Oct 30 22:13:56.937330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:56.937290    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-09eabe-wn7m7"]
Oct 30 22:13:57.220329 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.220229    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31334a4f-e7ba-47bc-b0d6-ce0c98acea82" path="/var/lib/kubelet/pods/31334a4f-e7ba-47bc-b0d6-ce0c98acea82/volumes"
Oct 30 22:13:57.840510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.840470    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"]
Oct 30 22:13:57.841076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841062    6695 cpu_manager.go:414] "containerMap: removing container" podUID="77679b84-bbb2-4efe-b913-9a9b676f7ac3" containerName="revoke-keys"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841077    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77679b84-bbb2-4efe-b913-9a9b676f7ac3" containerName="revoke-keys"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841098    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841104    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841118    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841122    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841129    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841133    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841149    6695 cpu_manager.go:414] "containerMap: removing container" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerName="payload-pre-processing"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841153    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerName="payload-pre-processing"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841162    6695 cpu_manager.go:414] "containerMap: removing container" podUID="31334a4f-e7ba-47bc-b0d6-ce0c98acea82" containerName="revoke-keys"
Oct 30 22:13:57.841167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841167    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31334a4f-e7ba-47bc-b0d6-ce0c98acea82" containerName="revoke-keys"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841174    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c8bec463-2506-438a-bc34-c177ced34fea" containerName="revoke-keys"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841179    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bec463-2506-438a-bc34-c177ced34fea" containerName="revoke-keys"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841262    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="77679b84-bbb2-4efe-b913-9a9b676f7ac3" containerName="revoke-keys"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841271    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841278    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841284    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c656625-0f5d-422f-a718-c447c368fcf6" containerName="istio-proxy"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841291    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="69963d5e-752a-4fe1-835e-23f0a83a58a3" containerName="payload-pre-processing"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841298    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="31334a4f-e7ba-47bc-b0d6-ce0c98acea82" containerName="revoke-keys"
Oct 30 22:13:57.841521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.841303    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bec463-2506-438a-bc34-c177ced34fea" containerName="revoke-keys"
Oct 30 22:13:57.844839 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.844822    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:57.863271 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.863238    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"]
Oct 30 22:13:57.871642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.870529    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:57.871642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.870532    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:57.914363 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.914326    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:57.914547 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.914487    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:13:57.994522 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.994481    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:57.994964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.994645    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr89l\" (UniqueName: \"kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:57.994964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:57.994861    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.095928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.095833    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.095928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.095888    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.096183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.096061    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr89l\" (UniqueName: \"kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.096700 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.096677    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.098346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.098325    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.133941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.133904    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr89l\" (UniqueName: \"kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l\") pod \"maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.159121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.159089    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:13:58.400284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.400197    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"]
Oct 30 22:13:58.400502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.400473    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy" containerID="cri-o://91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc" gracePeriod=30
Oct 30 22:13:58.465235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.465197    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"]
Oct 30 22:13:58.468900 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:13:58.468869    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efd54e0_3ef3_428e_8262_681f802619fc.slice/crio-324f0c66a7d92d5ccec7671f22c203e500356def043785c83e9a33c7a2769d5b WatchSource:0}: Error finding container 324f0c66a7d92d5ccec7671f22c203e500356def043785c83e9a33c7a2769d5b: Status 404 returned error can't find the container with id 324f0c66a7d92d5ccec7671f22c203e500356def043785c83e9a33c7a2769d5b
Oct 30 22:13:58.784288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.784249    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2" event={"ID":"5efd54e0-3ef3-428e-8262-681f802619fc","Type":"ContainerStarted","Data":"324f0c66a7d92d5ccec7671f22c203e500356def043785c83e9a33c7a2769d5b"}
Oct 30 22:13:58.951784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.951749    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"]
Oct 30 22:13:58.952384 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.952371    6695 cpu_manager.go:414] "containerMap: removing container" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy"
Oct 30 22:13:58.952432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.952386    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy"
Oct 30 22:13:58.952492 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.952482    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe820873-6dc6-44a7-8813-6902c183c856" containerName="istio-proxy"
Oct 30 22:13:58.956069 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.956049    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:58.974538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:58.974507    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"]
Oct 30 22:13:59.024011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.023979    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.024421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.024096    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlbv\" (UniqueName: \"kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.024421 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.024138    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.124969 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.124929    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.125147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.125029    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlbv\" (UniqueName: \"kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.125147 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.125070    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.125780 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.125757    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.127494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.127475    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.146160 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.146130    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlbv\" (UniqueName: \"kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv\") pod \"maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.269665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.269629    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:13:59.583682 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.583656    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"]
Oct 30 22:13:59.586596 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:13:59.586559    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1276b18_fdb7_47af_af88_de462fca0326.slice/crio-daacf4683ddfe6fffc922f867642172aebd436d58916141985a2ff3e89df15a4 WatchSource:0}: Error finding container daacf4683ddfe6fffc922f867642172aebd436d58916141985a2ff3e89df15a4: Status 404 returned error can't find the container with id daacf4683ddfe6fffc922f867642172aebd436d58916141985a2ff3e89df15a4
Oct 30 22:13:59.794137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.794097    6695 generic.go:356] "Generic (PLEG): container finished" podID="5efd54e0-3ef3-428e-8262-681f802619fc" containerID="6df49271cf6955555e02e8205611d36db72ec98956f9efb845407a85fdcea7bc" exitCode=0
Oct 30 22:13:59.794321 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.794180    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2" event={"ID":"5efd54e0-3ef3-428e-8262-681f802619fc","Type":"ContainerDied","Data":"6df49271cf6955555e02e8205611d36db72ec98956f9efb845407a85fdcea7bc"}
Oct 30 22:13:59.795584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:13:59.795561    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5" event={"ID":"f1276b18-fdb7-47af-af88-de462fca0326","Type":"ContainerStarted","Data":"daacf4683ddfe6fffc922f867642172aebd436d58916141985a2ff3e89df15a4"}
Oct 30 22:14:00.804297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:00.804261    6695 generic.go:356] "Generic (PLEG): container finished" podID="f1276b18-fdb7-47af-af88-de462fca0326" containerID="40da4232ef86f68af99377670f819f1d33cfc3e5989aa94439db9323d72b43bd" exitCode=0
Oct 30 22:14:00.804836 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:00.804353    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5" event={"ID":"f1276b18-fdb7-47af-af88-de462fca0326","Type":"ContainerDied","Data":"40da4232ef86f68af99377670f819f1d33cfc3e5989aa94439db9323d72b43bd"}
Oct 30 22:14:00.963175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:00.963148    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:14:01.047774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.047744    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca\") pod \"5efd54e0-3ef3-428e-8262-681f802619fc\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") "
Oct 30 22:14:01.047964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.047931    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca\") pod \"5efd54e0-3ef3-428e-8262-681f802619fc\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") "
Oct 30 22:14:01.048038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.048021    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l\" (UniqueName: \"kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l\") pod \"5efd54e0-3ef3-428e-8262-681f802619fc\" (UID: \"5efd54e0-3ef3-428e-8262-681f802619fc\") "
Oct 30 22:14:01.048355 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.048328    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca" pod "5efd54e0-3ef3-428e-8262-681f802619fc" (UID: "5efd54e0-3ef3-428e-8262-681f802619fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:01.049975 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.049942    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca" pod "5efd54e0-3ef3-428e-8262-681f802619fc" (UID: "5efd54e0-3ef3-428e-8262-681f802619fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:01.050098 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.050073    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l" pod "5efd54e0-3ef3-428e-8262-681f802619fc" (UID: "5efd54e0-3ef3-428e-8262-681f802619fc"). InnerVolumeSpecName "kube-api-access-lr89l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:01.078406 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:01.078366    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:14:01.149713 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.149683    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:01.149713 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.149710    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efd54e0-3ef3-428e-8262-681f802619fc-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:01.149908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.149720    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr89l\" (UniqueName: \"kubernetes.io/projected/5efd54e0-3ef3-428e-8262-681f802619fc-kube-api-access-lr89l\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:01.572137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.572099    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:01.802648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.802581    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:01.813165 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.813131    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"
Oct 30 22:14:01.813584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.813168    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2" event={"ID":"5efd54e0-3ef3-428e-8262-681f802619fc","Type":"ContainerDied","Data":"324f0c66a7d92d5ccec7671f22c203e500356def043785c83e9a33c7a2769d5b"}
Oct 30 22:14:01.813584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.813217    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="324f0c66a7d92d5ccec7671f22c203e500356def043785c83e9a33c7a2769d5b"
Oct 30 22:14:01.983678 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:01.983651    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:14:02.060950 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.060912    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv\" (UniqueName: \"kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv\") pod \"f1276b18-fdb7-47af-af88-de462fca0326\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") "
Oct 30 22:14:02.061188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.061012    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca\") pod \"f1276b18-fdb7-47af-af88-de462fca0326\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") "
Oct 30 22:14:02.061268 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.061254    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca\") pod \"f1276b18-fdb7-47af-af88-de462fca0326\" (UID: \"f1276b18-fdb7-47af-af88-de462fca0326\") "
Oct 30 22:14:02.061838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.061806    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca" pod "f1276b18-fdb7-47af-af88-de462fca0326" (UID: "f1276b18-fdb7-47af-af88-de462fca0326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:02.063337 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.063312    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv" pod "f1276b18-fdb7-47af-af88-de462fca0326" (UID: "f1276b18-fdb7-47af-af88-de462fca0326"). InnerVolumeSpecName "kube-api-access-jzlbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:02.063773 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.063756    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca" pod "f1276b18-fdb7-47af-af88-de462fca0326" (UID: "f1276b18-fdb7-47af-af88-de462fca0326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:02.162743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.162663    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:02.162743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.162693    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzlbv\" (UniqueName: \"kubernetes.io/projected/f1276b18-fdb7-47af-af88-de462fca0326-kube-api-access-jzlbv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:02.162743 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.162703    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1276b18-fdb7-47af-af88-de462fca0326-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:02.517514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.517415    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:02.517514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.517442    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:02.822845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.822808    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"
Oct 30 22:14:02.823244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.822846    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5" event={"ID":"f1276b18-fdb7-47af-af88-de462fca0326","Type":"ContainerDied","Data":"daacf4683ddfe6fffc922f867642172aebd436d58916141985a2ff3e89df15a4"}
Oct 30 22:14:02.823244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.822899    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daacf4683ddfe6fffc922f867642172aebd436d58916141985a2ff3e89df15a4"
Oct 30 22:14:02.874904 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:02.874862    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:03.433684 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:03.433651    6695 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6d8f36_f381_4ebd_a5a7_912a93c49842.slice/crio-91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:14:03.686346 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.686278    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 22:14:03.781622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781572    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.781844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781641    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.781844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781676    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.781844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781702    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.781844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781727    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.781844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781792    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.781844 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.781841    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.782191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782008    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.782191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782042    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo\") pod \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\" (UID: \"bb6d8f36-f381-4ebd-a5a7-912a93c49842\") "
Oct 30 22:14:03.782191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782047    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:03.782191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782063    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:03.782191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782108    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:03.782536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782454    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:03.782536 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782518    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:03.782731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782532    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.782731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782554    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.782731 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.782567    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.784517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.784491    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:03.784697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.784678    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:03.784814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.784797    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:03.784889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.784812    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x" pod "bb6d8f36-f381-4ebd-a5a7-912a93c49842" (UID: "bb6d8f36-f381-4ebd-a5a7-912a93c49842"). InnerVolumeSpecName "kube-api-access-tvt5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:03.832131 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.832091    6695 generic.go:356] "Generic (PLEG): container finished" podID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerID="91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc" exitCode=0
Oct 30 22:14:03.832577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.832136    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" event={"ID":"bb6d8f36-f381-4ebd-a5a7-912a93c49842","Type":"ContainerDied","Data":"91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc"}
Oct 30 22:14:03.832577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.832145    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"
Oct 30 22:14:03.832577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.832185    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x" event={"ID":"bb6d8f36-f381-4ebd-a5a7-912a93c49842","Type":"ContainerDied","Data":"a6e17784250dca1a151c2582bbee50321708bb82c27361c7122a4355905a8a1c"}
Oct 30 22:14:03.832577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.832209    6695 scope.go:122] "RemoveContainer" containerID="91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc"
Oct 30 22:14:03.843787 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.843769    6695 scope.go:122] "RemoveContainer" containerID="921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"
Oct 30 22:14:03.853569 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.853547    6695 scope.go:122] "RemoveContainer" containerID="91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc"
Oct 30 22:14:03.853888 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:03.853864    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc\": container with ID starting with 91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc not found: ID does not exist" containerID="91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc"
Oct 30 22:14:03.853984 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.853894    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc"} err="failed to get container status \"91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc\": rpc error: code = NotFound desc = could not find container \"91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc\": container with ID starting with 91d3e451c620a45f156fafb77d768a5bd857f19daf41e522b3ec71df203ca5bc not found: ID does not exist"
Oct 30 22:14:03.853984 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.853911    6695 scope.go:122] "RemoveContainer" containerID="921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"
Oct 30 22:14:03.854187 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:03.854166    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670\": container with ID starting with 921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670 not found: ID does not exist" containerID="921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"
Oct 30 22:14:03.854243 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.854193    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670"} err="failed to get container status \"921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670\": rpc error: code = NotFound desc = could not find container \"921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670\": container with ID starting with 921d982a8352cf91ac5ba4af1272fceb4ce0a9384a62b1b0abc780260250e670 not found: ID does not exist"
Oct 30 22:14:03.883464 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883427    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"]
Oct 30 22:14:03.883819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883789    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.883819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883816    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.883923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883826    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.883923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883838    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.883923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883847    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvt5x\" (UniqueName: \"kubernetes.io/projected/bb6d8f36-f381-4ebd-a5a7-912a93c49842-kube-api-access-tvt5x\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.883923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.883856    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb6d8f36-f381-4ebd-a5a7-912a93c49842-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:03.893078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:03.893049    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-htt2x"]
Oct 30 22:14:05.069550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.069511    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"]
Oct 30 22:14:05.070439 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070416    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5efd54e0-3ef3-428e-8262-681f802619fc" containerName="revoke-keys"
Oct 30 22:14:05.070490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070443    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efd54e0-3ef3-428e-8262-681f802619fc" containerName="revoke-keys"
Oct 30 22:14:05.070490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070473    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy"
Oct 30 22:14:05.070490 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070481    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy"
Oct 30 22:14:05.070596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070507    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy"
Oct 30 22:14:05.070596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070516    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy"
Oct 30 22:14:05.070596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070552    6695 cpu_manager.go:414] "containerMap: removing container" podUID="f1276b18-fdb7-47af-af88-de462fca0326" containerName="revoke-keys"
Oct 30 22:14:05.070596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070560    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1276b18-fdb7-47af-af88-de462fca0326" containerName="revoke-keys"
Oct 30 22:14:05.070783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070682    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy"
Oct 30 22:14:05.070783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070693    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" containerName="istio-proxy"
Oct 30 22:14:05.070783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070704    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1276b18-fdb7-47af-af88-de462fca0326" containerName="revoke-keys"
Oct 30 22:14:05.070783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.070713    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efd54e0-3ef3-428e-8262-681f802619fc" containerName="revoke-keys"
Oct 30 22:14:05.075487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.075469    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.092213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.092190    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"]
Oct 30 22:14:05.098617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.098575    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.098828 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.098806    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp57f\" (UniqueName: \"kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.098936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.098851    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.100131 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.100109    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:14:05.101501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.101477    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:05.101634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.101507    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:14:05.101634 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.101515    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:14:05.200527 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.200490    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp57f\" (UniqueName: \"kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.200774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.200536    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.200774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.200586    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.201302 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.201265    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.203087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.203062    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.215260 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.215228    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6d8f36-f381-4ebd-a5a7-912a93c49842" path="/var/lib/kubelet/pods/bb6d8f36-f381-4ebd-a5a7-912a93c49842/volumes"
Oct 30 22:14:05.219559 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.219533    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp57f\" (UniqueName: \"kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.389465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.389385    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:05.698819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.698733    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"]
Oct 30 22:14:05.704296 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:05.704254    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd4c4f2_fd75_4fd8_bcd0_617604eab917.slice/crio-494895b244d43d2fccdf24a43135a35ea3ea86ea7259dab64bea240a794ac7f3 WatchSource:0}: Error finding container 494895b244d43d2fccdf24a43135a35ea3ea86ea7259dab64bea240a794ac7f3: Status 404 returned error can't find the container with id 494895b244d43d2fccdf24a43135a35ea3ea86ea7259dab64bea240a794ac7f3
Oct 30 22:14:05.851258 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:05.851221    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j" event={"ID":"8cd4c4f2-fd75-4fd8-bcd0-617604eab917","Type":"ContainerStarted","Data":"494895b244d43d2fccdf24a43135a35ea3ea86ea7259dab64bea240a794ac7f3"}
Oct 30 22:14:06.860373 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:06.860337    6695 generic.go:356] "Generic (PLEG): container finished" podID="8cd4c4f2-fd75-4fd8-bcd0-617604eab917" containerID="0e4bcd73b77facf68b6aa02de484bb3014b9b9f39efa614db08e5a5d5ee65fc3" exitCode=0
Oct 30 22:14:06.860921 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:06.860431    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j" event={"ID":"8cd4c4f2-fd75-4fd8-bcd0-617604eab917","Type":"ContainerDied","Data":"0e4bcd73b77facf68b6aa02de484bb3014b9b9f39efa614db08e5a5d5ee65fc3"}
Oct 30 22:14:07.449588 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.449559    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 22:14:07.496427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.496399    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 22:14:07.531253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531225    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume\") pod \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") "
Oct 30 22:14:07.531443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531263    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q\" (UniqueName: \"kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q\") pod \"bebf10d2-094f-4568-8f42-1c15cb8b786b\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") "
Oct 30 22:14:07.531443 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw\" (UniqueName: \"kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw\") pod \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\" (UID: \"a99522c1-9a6c-4985-bf28-1a45619b9dd5\") "
Oct 30 22:14:07.531573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531444    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume\") pod \"bebf10d2-094f-4568-8f42-1c15cb8b786b\" (UID: \"bebf10d2-094f-4568-8f42-1c15cb8b786b\") "
Oct 30 22:14:07.531723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531698    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume" pod "a99522c1-9a6c-4985-bf28-1a45619b9dd5" (UID: "a99522c1-9a6c-4985-bf28-1a45619b9dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:07.531899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531879    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99522c1-9a6c-4985-bf28-1a45619b9dd5-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:07.531972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.531898    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume" pod "bebf10d2-094f-4568-8f42-1c15cb8b786b" (UID: "bebf10d2-094f-4568-8f42-1c15cb8b786b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:07.533783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.533759    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw" pod "a99522c1-9a6c-4985-bf28-1a45619b9dd5" (UID: "a99522c1-9a6c-4985-bf28-1a45619b9dd5"). InnerVolumeSpecName "kube-api-access-5dbkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:07.533879 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.533845    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q" pod "bebf10d2-094f-4568-8f42-1c15cb8b786b" (UID: "bebf10d2-094f-4568-8f42-1c15cb8b786b"). InnerVolumeSpecName "kube-api-access-sfn8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:07.633478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.633440    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbkw\" (UniqueName: \"kubernetes.io/projected/a99522c1-9a6c-4985-bf28-1a45619b9dd5-kube-api-access-5dbkw\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:07.633478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.633473    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebf10d2-094f-4568-8f42-1c15cb8b786b-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:07.633478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.633482    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn8q\" (UniqueName: \"kubernetes.io/projected/bebf10d2-094f-4568-8f42-1c15cb8b786b-kube-api-access-sfn8q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:07.869446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.869408    6695 generic.go:356] "Generic (PLEG): container finished" podID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerID="e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31" exitCode=0
Oct 30 22:14:07.869988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.869467    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"
Oct 30 22:14:07.869988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.869495    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" event={"ID":"bebf10d2-094f-4568-8f42-1c15cb8b786b","Type":"ContainerDied","Data":"e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31"}
Oct 30 22:14:07.869988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.869551    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m" event={"ID":"bebf10d2-094f-4568-8f42-1c15cb8b786b","Type":"ContainerDied","Data":"72f11867ea751a04febad6f662adac873b0817a39597f18879df924f79562c7e"}
Oct 30 22:14:07.869988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.869576    6695 scope.go:122] "RemoveContainer" containerID="e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31"
Oct 30 22:14:07.871513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.871469    6695 generic.go:356] "Generic (PLEG): container finished" podID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerID="a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6" exitCode=0
Oct 30 22:14:07.871742 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.871510    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" event={"ID":"a99522c1-9a6c-4985-bf28-1a45619b9dd5","Type":"ContainerDied","Data":"a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6"}
Oct 30 22:14:07.871742 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.871558    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh" event={"ID":"a99522c1-9a6c-4985-bf28-1a45619b9dd5","Type":"ContainerDied","Data":"dff843568f37268f5a01a8c0a579bed4e07bb3a86a3c3bcef0f630b7bd69015f"}
Oct 30 22:14:07.872853 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.871969    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"
Oct 30 22:14:07.883004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.882978    6695 scope.go:122] "RemoveContainer" containerID="e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31"
Oct 30 22:14:07.883371 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:07.883351    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31\": container with ID starting with e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31 not found: ID does not exist" containerID="e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31"
Oct 30 22:14:07.883470 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.883382    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31"} err="failed to get container status \"e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31\": rpc error: code = NotFound desc = could not find container \"e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31\": container with ID starting with e405a18b675ea85425488a0f4d7fa847233bc89e54497ffdb337f2c9f1d1ec31 not found: ID does not exist"
Oct 30 22:14:07.883470 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.883404    6695 scope.go:122] "RemoveContainer" containerID="a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6"
Oct 30 22:14:07.894047 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.894024    6695 scope.go:122] "RemoveContainer" containerID="a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6"
Oct 30 22:14:07.894361 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:07.894336    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6\": container with ID starting with a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6 not found: ID does not exist" containerID="a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6"
Oct 30 22:14:07.894453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.894367    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6"} err="failed to get container status \"a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6\": rpc error: code = NotFound desc = could not find container \"a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6\": container with ID starting with a7d3addd14ee7333ea2b27f90341e525580542a273fba52dd95b3b9ef19213d6 not found: ID does not exist"
Oct 30 22:14:07.927109 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.927074    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"]
Oct 30 22:14:07.939539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.939504    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5d3601-59cfb8487c-75l8m"]
Oct 30 22:14:07.983878 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.983845    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"]
Oct 30 22:14:07.990199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:07.990167    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5d3601-7fbdddb5cf-dn9fh"]
Oct 30 22:14:08.077995 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.077961    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:08.142596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.142562    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca\") pod \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") "
Oct 30 22:14:08.142793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.142679    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca\") pod \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") "
Oct 30 22:14:08.142793 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.142760    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f\" (UniqueName: \"kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f\") pod \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\" (UID: \"8cd4c4f2-fd75-4fd8-bcd0-617604eab917\") "
Oct 30 22:14:08.143118 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.143091    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca" pod "8cd4c4f2-fd75-4fd8-bcd0-617604eab917" (UID: "8cd4c4f2-fd75-4fd8-bcd0-617604eab917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:08.143708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.143681    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.145471 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.145418    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f" pod "8cd4c4f2-fd75-4fd8-bcd0-617604eab917" (UID: "8cd4c4f2-fd75-4fd8-bcd0-617604eab917"). InnerVolumeSpecName "kube-api-access-lp57f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:08.150238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.150214    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca" pod "8cd4c4f2-fd75-4fd8-bcd0-617604eab917" (UID: "8cd4c4f2-fd75-4fd8-bcd0-617604eab917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:08.199479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.199442    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 22:14:08.244677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.244569    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97\" (UniqueName: \"kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97\") pod \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") "
Oct 30 22:14:08.244677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.244638    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume\") pod \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\" (UID: \"e1b52dd2-cbbe-47bb-ab7d-4a259c376588\") "
Oct 30 22:14:08.245245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.245084    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.245245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.245107    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp57f\" (UniqueName: \"kubernetes.io/projected/8cd4c4f2-fd75-4fd8-bcd0-617604eab917-kube-api-access-lp57f\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.245389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.245255    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume" pod "e1b52dd2-cbbe-47bb-ab7d-4a259c376588" (UID: "e1b52dd2-cbbe-47bb-ab7d-4a259c376588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:08.247782 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.247751    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97" pod "e1b52dd2-cbbe-47bb-ab7d-4a259c376588" (UID: "e1b52dd2-cbbe-47bb-ab7d-4a259c376588"). InnerVolumeSpecName "kube-api-access-dwz97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:08.283748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.283714    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 22:14:08.312635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.312585    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"]
Oct 30 22:14:08.345479 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.345419    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj\" (UniqueName: \"kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj\") pod \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") "
Oct 30 22:14:08.345768 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.345668    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume\") pod \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\" (UID: \"c6194418-0e26-4d4e-bd08-d44266b3ca6e\") "
Oct 30 22:14:08.346139 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.346115    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwz97\" (UniqueName: \"kubernetes.io/projected/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-kube-api-access-dwz97\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.349309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.346351    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b52dd2-cbbe-47bb-ab7d-4a259c376588-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.349309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.346594    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume" pod "c6194418-0e26-4d4e-bd08-d44266b3ca6e" (UID: "c6194418-0e26-4d4e-bd08-d44266b3ca6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:08.349514 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.349399    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj" pod "c6194418-0e26-4d4e-bd08-d44266b3ca6e" (UID: "c6194418-0e26-4d4e-bd08-d44266b3ca6e"). InnerVolumeSpecName "kube-api-access-7mjhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:08.350413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.350389    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"]
Oct 30 22:14:08.447805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.447727    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjhj\" (UniqueName: \"kubernetes.io/projected/c6194418-0e26-4d4e-bd08-d44266b3ca6e-kube-api-access-7mjhj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.447805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.447759    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6194418-0e26-4d4e-bd08-d44266b3ca6e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.880868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.880829    6695 generic.go:356] "Generic (PLEG): container finished" podID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerID="02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27" exitCode=0
Oct 30 22:14:08.881298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.880882    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"
Oct 30 22:14:08.881298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.880907    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" event={"ID":"c6194418-0e26-4d4e-bd08-d44266b3ca6e","Type":"ContainerDied","Data":"02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27"}
Oct 30 22:14:08.881298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.880960    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw" event={"ID":"c6194418-0e26-4d4e-bd08-d44266b3ca6e","Type":"ContainerDied","Data":"0889ad19648a4f4b9dfc213569acd4c4e1587549e20bcbf1b0a89998071d279d"}
Oct 30 22:14:08.881298 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.880983    6695 scope.go:122] "RemoveContainer" containerID="02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27"
Oct 30 22:14:08.882584 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.882560    6695 generic.go:356] "Generic (PLEG): container finished" podID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerID="99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95" exitCode=0
Oct 30 22:14:08.882723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.882640    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" event={"ID":"e1b52dd2-cbbe-47bb-ab7d-4a259c376588","Type":"ContainerDied","Data":"99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95"}
Oct 30 22:14:08.882723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.882697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z" event={"ID":"e1b52dd2-cbbe-47bb-ab7d-4a259c376588","Type":"ContainerDied","Data":"489281b0d3ec07c1277503a93d850ba8286f0946ff657b2d397a7bf9dc18685b"}
Oct 30 22:14:08.882723 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.882651    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"
Oct 30 22:14:08.884676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.884650    6695 generic.go:356] "Generic (PLEG): container finished" podID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerID="52de8f4e196b59d44947f5f309264cbe7c286df7a68bafff74ab9310395e9c4c" exitCode=0
Oct 30 22:14:08.884803 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.884783    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" event={"ID":"3406f3a1-57d0-418a-99b0-2ef323fe49d6","Type":"ContainerDied","Data":"52de8f4e196b59d44947f5f309264cbe7c286df7a68bafff74ab9310395e9c4c"}
Oct 30 22:14:08.889136 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.889115    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j" event={"ID":"8cd4c4f2-fd75-4fd8-bcd0-617604eab917","Type":"ContainerDied","Data":"494895b244d43d2fccdf24a43135a35ea3ea86ea7259dab64bea240a794ac7f3"}
Oct 30 22:14:08.889253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.889158    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="494895b244d43d2fccdf24a43135a35ea3ea86ea7259dab64bea240a794ac7f3"
Oct 30 22:14:08.889253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.889211    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"
Oct 30 22:14:08.896381 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.896362    6695 scope.go:122] "RemoveContainer" containerID="02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27"
Oct 30 22:14:08.896829 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:08.896803    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27\": container with ID starting with 02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27 not found: ID does not exist" containerID="02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27"
Oct 30 22:14:08.896917 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.896839    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27"} err="failed to get container status \"02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27\": rpc error: code = NotFound desc = could not find container \"02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27\": container with ID starting with 02fd444aed2318ab8f4eef113cd94efd9cd5b92e86bab82d01b482e56ace4f27 not found: ID does not exist"
Oct 30 22:14:08.896917 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.896859    6695 scope.go:122] "RemoveContainer" containerID="99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95"
Oct 30 22:14:08.910417 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.910393    6695 scope.go:122] "RemoveContainer" containerID="99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95"
Oct 30 22:14:08.910797 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:08.910757    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95\": container with ID starting with 99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95 not found: ID does not exist" containerID="99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95"
Oct 30 22:14:08.910894 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:08.910805    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95"} err="failed to get container status \"99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95\": rpc error: code = NotFound desc = could not find container \"99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95\": container with ID starting with 99bec7ab421ae3efc5e355e2a347d280a13ffa50171704d66d54ce7633acaf95 not found: ID does not exist"
Oct 30 22:14:09.022913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.020941    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"]
Oct 30 22:14:09.023409 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.023370    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-09eabe-5b84f9475f-lhfkw"]
Oct 30 22:14:09.029202 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.029174    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"]
Oct 30 22:14:09.033685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.033655    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-09eabe-59cfcdbb59-ch99z"]
Oct 30 22:14:09.049560 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.049535    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 22:14:09.158123 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.158087    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj\" (UniqueName: \"kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj\") pod \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") "
Oct 30 22:14:09.158303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.158239    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume\") pod \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\" (UID: \"3406f3a1-57d0-418a-99b0-2ef323fe49d6\") "
Oct 30 22:14:09.158617 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.158581    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume" pod "3406f3a1-57d0-418a-99b0-2ef323fe49d6" (UID: "3406f3a1-57d0-418a-99b0-2ef323fe49d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:09.160160 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.160136    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj" pod "3406f3a1-57d0-418a-99b0-2ef323fe49d6" (UID: "3406f3a1-57d0-418a-99b0-2ef323fe49d6"). InnerVolumeSpecName "kube-api-access-9vwsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:09.216015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.215975    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" path="/var/lib/kubelet/pods/a99522c1-9a6c-4985-bf28-1a45619b9dd5/volumes"
Oct 30 22:14:09.216630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.216586    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" path="/var/lib/kubelet/pods/bebf10d2-094f-4568-8f42-1c15cb8b786b/volumes"
Oct 30 22:14:09.217226 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.217204    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" path="/var/lib/kubelet/pods/c6194418-0e26-4d4e-bd08-d44266b3ca6e/volumes"
Oct 30 22:14:09.217762 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.217744    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" path="/var/lib/kubelet/pods/e1b52dd2-cbbe-47bb-ab7d-4a259c376588/volumes"
Oct 30 22:14:09.259921 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.259892    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwsj\" (UniqueName: \"kubernetes.io/projected/3406f3a1-57d0-418a-99b0-2ef323fe49d6-kube-api-access-9vwsj\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.259921 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.259916    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3406f3a1-57d0-418a-99b0-2ef323fe49d6-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.489619 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.489529    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"]
Oct 30 22:14:09.521693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.521659    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"]
Oct 30 22:14:09.899462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.899433    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"
Oct 30 22:14:09.899462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.899452    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc" event={"ID":"3406f3a1-57d0-418a-99b0-2ef323fe49d6","Type":"ContainerDied","Data":"ec3bf4dfd09f74b880e6ae31842d78bdc52041d1c3157c4d216830003d628022"}
Oct 30 22:14:09.900108 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.899539    6695 scope.go:122] "RemoveContainer" containerID="52de8f4e196b59d44947f5f309264cbe7c286df7a68bafff74ab9310395e9c4c"
Oct 30 22:14:09.952153 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.952116    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"]
Oct 30 22:14:09.958208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:09.958174    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-805edc-66887bd4bf-ts8vc"]
Oct 30 22:14:11.226979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:11.226942    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" path="/var/lib/kubelet/pods/3406f3a1-57d0-418a-99b0-2ef323fe49d6/volumes"
Oct 30 22:14:11.664525 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:11.664483    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"]
Oct 30 22:14:11.674087 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:11.674055    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9722ac-dsmg5"]
Oct 30 22:14:12.170625 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:12.170571    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"]
Oct 30 22:14:12.181831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:12.181798    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ed0866-57jv2"]
Oct 30 22:14:13.214593 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:13.214556    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efd54e0-3ef3-428e-8262-681f802619fc" path="/var/lib/kubelet/pods/5efd54e0-3ef3-428e-8262-681f802619fc/volumes"
Oct 30 22:14:13.215039 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:13.215021    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1276b18-fdb7-47af-af88-de462fca0326" path="/var/lib/kubelet/pods/f1276b18-fdb7-47af-af88-de462fca0326/volumes"
Oct 30 22:14:13.313511 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:13.313453    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" podUID="a0566572-3228-448b-8636-249effd583d4" containerName="payload-processing" containerID="cri-o://94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82" gracePeriod=25
Oct 30 22:14:13.350763 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:13.350718    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerName="payload-pre-processing" containerID="cri-o://abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854" gracePeriod=25
Oct 30 22:14:14.490114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:14.490053    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerName="payload-processing" containerID="cri-o://80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91" gracePeriod=25
Oct 30 22:14:14.522183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:14.522143    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerName="payload-pre-processing" containerID="cri-o://f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48" gracePeriod=25
Oct 30 22:14:15.544783 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:15.544744    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"]
Oct 30 22:14:15.582506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:15.582467    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"]
Oct 30 22:14:17.706224 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:17.706185    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"]
Oct 30 22:14:17.717877 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:17.717847    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-bfabe1-xx88j"]
Oct 30 22:14:19.213523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:19.213488    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd4c4f2-fd75-4fd8-bcd0-617604eab917" path="/var/lib/kubelet/pods/8cd4c4f2-fd75-4fd8-bcd0-617604eab917/volumes"
Oct 30 22:14:19.723550 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:19.723515    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"]
Oct 30 22:14:19.723784 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:19.723760    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy" containerID="cri-o://14e4bbab468d541eb545ea9379d4bddb6df46270ddf02285f7c1623e821f65e3" gracePeriod=30
Oct 30 22:14:19.768998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:19.768956    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" podUID="a0566572-3228-448b-8636-249effd583d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:19.768998 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:19.768981    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:20.315637 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.315563    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"]
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316303    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerName="payload-pre-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316327    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerName="payload-pre-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316339    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316345    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316361    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316366    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316391    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerName="payload-pre-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316399    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerName="payload-pre-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316419    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316426    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316444    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8cd4c4f2-fd75-4fd8-bcd0-617604eab917" containerName="revoke-keys"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316451    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd4c4f2-fd75-4fd8-bcd0-617604eab917" containerName="revoke-keys"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316571    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebf10d2-094f-4568-8f42-1c15cb8b786b" containerName="payload-processing"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316587    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd4c4f2-fd75-4fd8-bcd0-617604eab917" containerName="revoke-keys"
Oct 30 22:14:20.317323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.316599    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6194418-0e26-4d4e-bd08-d44266b3ca6e" containerName="payload-pre-processing"
Oct 30 22:14:20.318297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.317353    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1b52dd2-cbbe-47bb-ab7d-4a259c376588" containerName="payload-processing"
Oct 30 22:14:20.318297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.317676    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3406f3a1-57d0-418a-99b0-2ef323fe49d6" containerName="payload-processing"
Oct 30 22:14:20.318297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.318011    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99522c1-9a6c-4985-bf28-1a45619b9dd5" containerName="payload-pre-processing"
Oct 30 22:14:20.325857 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.325836    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.343403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.343293    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"]
Oct 30 22:14:20.350417 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.350390    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:14:20.350533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.350439    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:20.350533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.350501    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:14:20.350680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.350557    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:14:20.423152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.423109    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.423313 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.423166    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbxk\" (UniqueName: \"kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.423378 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.423348    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.524034 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.523994    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.524254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.524101    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.524254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.524136    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbxk\" (UniqueName: \"kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.524801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.524773    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.526540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.526511    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.546072 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.546042    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerName="payload-processing" containerID="cri-o://37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119" gracePeriod=25
Oct 30 22:14:20.561761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.561739    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbxk\" (UniqueName: \"kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.582876 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.582767    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerName="payload-pre-processing" containerID="cri-o://657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960" gracePeriod=25
Oct 30 22:14:20.644932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.644904    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:20.896593 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.896514    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"]
Oct 30 22:14:20.902211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.902177    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:20.919268 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.919215    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"]
Oct 30 22:14:20.928820 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.928789    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:20.929004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.928979    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvx7\" (UniqueName: \"kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:20.929211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.929186    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:20.957356 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.957010    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"]
Oct 30 22:14:20.961401 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:20.961369    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c86752_4733_41d6_8473_eb703c91f331.slice/crio-f1f7e948bca8e1a719b9f51133bde3967dc063c54c026256770bf559f0bdf493 WatchSource:0}: Error finding container f1f7e948bca8e1a719b9f51133bde3967dc063c54c026256770bf559f0bdf493: Status 404 returned error can't find the container with id f1f7e948bca8e1a719b9f51133bde3967dc063c54c026256770bf559f0bdf493
Oct 30 22:14:20.988284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:20.988253    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv" event={"ID":"64c86752-4733-41d6-8473-eb703c91f331","Type":"ContainerStarted","Data":"f1f7e948bca8e1a719b9f51133bde3967dc063c54c026256770bf559f0bdf493"}
Oct 30 22:14:21.030264 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.030234    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.030418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.030292    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.030418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.030359    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvx7\" (UniqueName: \"kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.030972 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.030946    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.032687 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.032664    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.049968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.049945    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvx7\" (UniqueName: \"kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.217181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.217146    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:21.526007 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.525928    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"]
Oct 30 22:14:21.529098 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:21.529066    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f92ebe_680e_4226_90b3_039d49b79887.slice/crio-022f0ff85217de30ef58fba2aa2a48c21e378d38b051821ef7ff6c586f503987 WatchSource:0}: Error finding container 022f0ff85217de30ef58fba2aa2a48c21e378d38b051821ef7ff6c586f503987: Status 404 returned error can't find the container with id 022f0ff85217de30ef58fba2aa2a48c21e378d38b051821ef7ff6c586f503987
Oct 30 22:14:21.998173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.998131    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz" event={"ID":"71f92ebe-680e-4226-90b3-039d49b79887","Type":"ContainerStarted","Data":"0ba940874f8802be071600573964ef3e62ab33b880e9c4dd6147af6f87cb6f69"}
Oct 30 22:14:21.998330 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.998208    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz" event={"ID":"71f92ebe-680e-4226-90b3-039d49b79887","Type":"ContainerStarted","Data":"022f0ff85217de30ef58fba2aa2a48c21e378d38b051821ef7ff6c586f503987"}
Oct 30 22:14:21.999932 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.999900    6695 generic.go:356] "Generic (PLEG): container finished" podID="64c86752-4733-41d6-8473-eb703c91f331" containerID="6abbd9e5cd7796cccbec464ddbf66bd765e396775c2c904d6666017e377b72b6" exitCode=0
Oct 30 22:14:22.000068 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:21.999976    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv" event={"ID":"64c86752-4733-41d6-8473-eb703c91f331","Type":"ContainerDied","Data":"6abbd9e5cd7796cccbec464ddbf66bd765e396775c2c904d6666017e377b72b6"}
Oct 30 22:14:22.036886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:22.036822    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz" podStartSLOduration=2.036806549 podStartE2EDuration="2.036806549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:22.036304624 +0000 UTC m=+3844.958762923" watchObservedRunningTime="2026-10-30 22:14:22.036806549 +0000 UTC m=+3844.959264847"
Oct 30 22:14:23.009387 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.009346    6695 generic.go:356] "Generic (PLEG): container finished" podID="71f92ebe-680e-4226-90b3-039d49b79887" containerID="0ba940874f8802be071600573964ef3e62ab33b880e9c4dd6147af6f87cb6f69" exitCode=0
Oct 30 22:14:23.009826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.009435    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz" event={"ID":"71f92ebe-680e-4226-90b3-039d49b79887","Type":"ContainerDied","Data":"0ba940874f8802be071600573964ef3e62ab33b880e9c4dd6147af6f87cb6f69"}
Oct 30 22:14:23.197364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.197336    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:23.261841 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.261745    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca\") pod \"64c86752-4733-41d6-8473-eb703c91f331\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") "
Oct 30 22:14:23.262020 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.261929    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca\") pod \"64c86752-4733-41d6-8473-eb703c91f331\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") "
Oct 30 22:14:23.262020 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.261988    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk\" (UniqueName: \"kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk\") pod \"64c86752-4733-41d6-8473-eb703c91f331\" (UID: \"64c86752-4733-41d6-8473-eb703c91f331\") "
Oct 30 22:14:23.262151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.262108    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca" pod "64c86752-4733-41d6-8473-eb703c91f331" (UID: "64c86752-4733-41d6-8473-eb703c91f331"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:23.262420 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.262398    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c86752-4733-41d6-8473-eb703c91f331-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:23.264628 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.264573    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk" pod "64c86752-4733-41d6-8473-eb703c91f331" (UID: "64c86752-4733-41d6-8473-eb703c91f331"). InnerVolumeSpecName "kube-api-access-txbxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:23.264725 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.264649    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca" pod "64c86752-4733-41d6-8473-eb703c91f331" (UID: "64c86752-4733-41d6-8473-eb703c91f331"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:23.363786 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.363747    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c86752-4733-41d6-8473-eb703c91f331-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:23.363786 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:23.363778    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbxk\" (UniqueName: \"kubernetes.io/projected/64c86752-4733-41d6-8473-eb703c91f331-kube-api-access-txbxk\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:24.019526 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.019494    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"
Oct 30 22:14:24.019946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.019538    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv" event={"ID":"64c86752-4733-41d6-8473-eb703c91f331","Type":"ContainerDied","Data":"f1f7e948bca8e1a719b9f51133bde3967dc063c54c026256770bf559f0bdf493"}
Oct 30 22:14:24.019946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.019590    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1f7e948bca8e1a719b9f51133bde3967dc063c54c026256770bf559f0bdf493"
Oct 30 22:14:24.166198 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.166171    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:24.248144 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.248101    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:24.248326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.248183    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:24.273516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.273434    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca\") pod \"71f92ebe-680e-4226-90b3-039d49b79887\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") "
Oct 30 22:14:24.273690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.273590    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca\") pod \"71f92ebe-680e-4226-90b3-039d49b79887\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") "
Oct 30 22:14:24.273690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.273647    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7\" (UniqueName: \"kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7\") pod \"71f92ebe-680e-4226-90b3-039d49b79887\" (UID: \"71f92ebe-680e-4226-90b3-039d49b79887\") "
Oct 30 22:14:24.273840 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.273814    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca" pod "71f92ebe-680e-4226-90b3-039d49b79887" (UID: "71f92ebe-680e-4226-90b3-039d49b79887"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:24.274078 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.274057    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:24.275765 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.275737    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca" pod "71f92ebe-680e-4226-90b3-039d49b79887" (UID: "71f92ebe-680e-4226-90b3-039d49b79887"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:24.275891 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.275853    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7" pod "71f92ebe-680e-4226-90b3-039d49b79887" (UID: "71f92ebe-680e-4226-90b3-039d49b79887"). InnerVolumeSpecName "kube-api-access-npvx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:24.375735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.375697    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvx7\" (UniqueName: \"kubernetes.io/projected/71f92ebe-680e-4226-90b3-039d49b79887-kube-api-access-npvx7\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:24.375735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.375730    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f92ebe-680e-4226-90b3-039d49b79887-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:24.956939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.956904    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"]
Oct 30 22:14:24.957261 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:24.957180    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy" containerID="cri-o://b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c" gracePeriod=30
Oct 30 22:14:25.028936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.028902    6695 generic.go:356] "Generic (PLEG): container finished" podID="406109b5-5975-4c91-95e7-739fa653abbb" containerID="14e4bbab468d541eb545ea9379d4bddb6df46270ddf02285f7c1623e821f65e3" exitCode=0
Oct 30 22:14:25.029308 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.028975    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" event={"ID":"406109b5-5975-4c91-95e7-739fa653abbb","Type":"ContainerDied","Data":"14e4bbab468d541eb545ea9379d4bddb6df46270ddf02285f7c1623e821f65e3"}
Oct 30 22:14:25.029308 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.029039    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th" event={"ID":"406109b5-5975-4c91-95e7-739fa653abbb","Type":"ContainerDied","Data":"3c11752dbda83a3ba344865740e320b308457f8ff3e1d91a4124d0d1157e56c7"}
Oct 30 22:14:25.029308 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.029059    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c11752dbda83a3ba344865740e320b308457f8ff3e1d91a4124d0d1157e56c7"
Oct 30 22:14:25.029308 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.029073    6695 scope.go:122] "RemoveContainer" containerID="0d0271f20c5e60e937f781321e88ab1813cc84cda59f70822005e040b309b9e7"
Oct 30 22:14:25.030516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.030500    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"
Oct 30 22:14:25.030642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.030535    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz" event={"ID":"71f92ebe-680e-4226-90b3-039d49b79887","Type":"ContainerDied","Data":"022f0ff85217de30ef58fba2aa2a48c21e378d38b051821ef7ff6c586f503987"}
Oct 30 22:14:25.030642 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.030581    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="022f0ff85217de30ef58fba2aa2a48c21e378d38b051821ef7ff6c586f503987"
Oct 30 22:14:25.032502 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.032482    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 22:14:25.085720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.085684    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.085999 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.085972    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086205 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086190    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086322    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086509    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086637    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086778 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086736    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086788    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.086907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.086844    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs\") pod \"406109b5-5975-4c91-95e7-739fa653abbb\" (UID: \"406109b5-5975-4c91-95e7-739fa653abbb\") "
Oct 30 22:14:25.087483 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.087457    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.091031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.090184    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.091031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.090507    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.091031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.090681    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.091914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.091885    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:25.093188 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.093163    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "kube-api-access-fdrhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:25.103872 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.095535    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:25.103975 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.100003    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.104061 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.101936    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo" pod "406109b5-5975-4c91-95e7-739fa653abbb" (UID: "406109b5-5975-4c91-95e7-739fa653abbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:25.188066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188030    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188063    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188074    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188085    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrhc\" (UniqueName: \"kubernetes.io/projected/406109b5-5975-4c91-95e7-739fa653abbb-kube-api-access-fdrhc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188095    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406109b5-5975-4c91-95e7-739fa653abbb-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188103    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188113    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188120    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406109b5-5975-4c91-95e7-739fa653abbb-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.188316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.188128    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406109b5-5975-4c91-95e7-739fa653abbb-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.682947 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.682864    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"]
Oct 30 22:14:25.683427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683412    6695 cpu_manager.go:414] "containerMap: removing container" podUID="71f92ebe-680e-4226-90b3-039d49b79887" containerName="revoke-keys"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683429    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f92ebe-680e-4226-90b3-039d49b79887" containerName="revoke-keys"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683441    6695 cpu_manager.go:414] "containerMap: removing container" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683447    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683454    6695 cpu_manager.go:414] "containerMap: removing container" podUID="64c86752-4733-41d6-8473-eb703c91f331" containerName="revoke-keys"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683459    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c86752-4733-41d6-8473-eb703c91f331" containerName="revoke-keys"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683480    6695 cpu_manager.go:414] "containerMap: removing container" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy"
Oct 30 22:14:25.683494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683486    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy"
Oct 30 22:14:25.683794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683584    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f92ebe-680e-4226-90b3-039d49b79887" containerName="revoke-keys"
Oct 30 22:14:25.683794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683594    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy"
Oct 30 22:14:25.683794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.683613    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c86752-4733-41d6-8473-eb703c91f331" containerName="revoke-keys"
Oct 30 22:14:25.688109 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.688087    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.703896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.703753    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:14:25.703896 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.703850    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:14:25.707101 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.707075    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"]
Oct 30 22:14:25.791167 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.791126    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:25.791353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.791226    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:14:25.793881 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.793854    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbn4\" (UniqueName: \"kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.794004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.793978    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.794164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.794145    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.895107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.895073    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbn4\" (UniqueName: \"kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.895279 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.895143    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.895279 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.895238    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.896019 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.895994    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.897785 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.897761    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:25.914802 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:25.914776    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbn4\" (UniqueName: \"kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4\") pod \"maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:26.003026 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:26.002944    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:26.040336 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:26.040301    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"
Oct 30 22:14:26.130644 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:26.130593    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"]
Oct 30 22:14:26.138527 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:26.138496    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9722ac-openshift-default-5ffdc4d88f-4v5th"]
Oct 30 22:14:26.307862 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:26.307829    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"]
Oct 30 22:14:26.311020 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:26.310983    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3443d6a4_2241_48fe_ad9d_e3910403d27b.slice/crio-3b26738402fa01b955e84ee049fb6afb1fd2a1c96bc7bb2cd2e79b4628534fdf WatchSource:0}: Error finding container 3b26738402fa01b955e84ee049fb6afb1fd2a1c96bc7bb2cd2e79b4628534fdf: Status 404 returned error can't find the container with id 3b26738402fa01b955e84ee049fb6afb1fd2a1c96bc7bb2cd2e79b4628534fdf
Oct 30 22:14:27.048895 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:27.048848    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx" event={"ID":"3443d6a4-2241-48fe-ad9d-e3910403d27b","Type":"ContainerStarted","Data":"c19f0c288a9ff0f3ba47f60581142f5f59e3571c9b3385d8c5c3c0f5253e6822"}
Oct 30 22:14:27.049366 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:27.048945    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx" event={"ID":"3443d6a4-2241-48fe-ad9d-e3910403d27b","Type":"ContainerStarted","Data":"3b26738402fa01b955e84ee049fb6afb1fd2a1c96bc7bb2cd2e79b4628534fdf"}
Oct 30 22:14:27.094646 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:27.094569    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx" podStartSLOduration=2.094550525 podStartE2EDuration="2.094550525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:27.0921475 +0000 UTC m=+3850.014605799" watchObservedRunningTime="2026-10-30 22:14:27.094550525 +0000 UTC m=+3850.017008823"
Oct 30 22:14:27.215815 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:27.215779    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406109b5-5975-4c91-95e7-739fa653abbb" path="/var/lib/kubelet/pods/406109b5-5975-4c91-95e7-739fa653abbb/volumes"
Oct 30 22:14:27.299158 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:27.299078    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:27.299322 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:27.299185    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:28.057162 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:28.057124    6695 generic.go:356] "Generic (PLEG): container finished" podID="3443d6a4-2241-48fe-ad9d-e3910403d27b" containerID="c19f0c288a9ff0f3ba47f60581142f5f59e3571c9b3385d8c5c3c0f5253e6822" exitCode=0
Oct 30 22:14:28.057162 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:28.057163    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx" event={"ID":"3443d6a4-2241-48fe-ad9d-e3910403d27b","Type":"ContainerDied","Data":"c19f0c288a9ff0f3ba47f60581142f5f59e3571c9b3385d8c5c3c0f5253e6822"}
Oct 30 22:14:29.204203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.204178    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:29.334868 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.334785    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca\") pod \"3443d6a4-2241-48fe-ad9d-e3910403d27b\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") "
Oct 30 22:14:29.335028 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.334871    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca\") pod \"3443d6a4-2241-48fe-ad9d-e3910403d27b\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") "
Oct 30 22:14:29.335028 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.334940    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4\" (UniqueName: \"kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4\") pod \"3443d6a4-2241-48fe-ad9d-e3910403d27b\" (UID: \"3443d6a4-2241-48fe-ad9d-e3910403d27b\") "
Oct 30 22:14:29.335552 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.335217    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca" pod "3443d6a4-2241-48fe-ad9d-e3910403d27b" (UID: "3443d6a4-2241-48fe-ad9d-e3910403d27b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:29.335726 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.335569    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:29.337066 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.337044    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca" pod "3443d6a4-2241-48fe-ad9d-e3910403d27b" (UID: "3443d6a4-2241-48fe-ad9d-e3910403d27b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:29.337166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.337066    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4" pod "3443d6a4-2241-48fe-ad9d-e3910403d27b" (UID: "3443d6a4-2241-48fe-ad9d-e3910403d27b"). InnerVolumeSpecName "kube-api-access-mtbn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:29.436189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.436158    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3443d6a4-2241-48fe-ad9d-e3910403d27b-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:29.436189 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.436188    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtbn4\" (UniqueName: \"kubernetes.io/projected/3443d6a4-2241-48fe-ad9d-e3910403d27b-kube-api-access-mtbn4\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:29.599110 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.599025    6695 scope.go:122] "RemoveContainer" containerID="06cdc1c7aefc644af58a7d2bc7707640f6b980a172cf5be96f4fa7d3b1da9d32"
Oct 30 22:14:29.610659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.610631    6695 scope.go:122] "RemoveContainer" containerID="89abea0952c57da23d61c34bb8d0904f5d49799d71248e6289d850c95c5bc967"
Oct 30 22:14:29.619880 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.619861    6695 scope.go:122] "RemoveContainer" containerID="14e4bbab468d541eb545ea9379d4bddb6df46270ddf02285f7c1623e821f65e3"
Oct 30 22:14:29.629247 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.629220    6695 scope.go:122] "RemoveContainer" containerID="38734b380bfb52f36f7ad7577fe43e026ff3d705d7e74cb2f5ec2ed8db177740"
Oct 30 22:14:29.638848 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.638820    6695 scope.go:122] "RemoveContainer" containerID="74a6ed220be4e44354b7242a5f960c6e65357cedefaa3f335ee613c8fa0061f1"
Oct 30 22:14:29.768874 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.768833    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" podUID="a0566572-3228-448b-8636-249effd583d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:29.769040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:29.768836    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:30.072526 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.072484    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx" event={"ID":"3443d6a4-2241-48fe-ad9d-e3910403d27b","Type":"ContainerDied","Data":"3b26738402fa01b955e84ee049fb6afb1fd2a1c96bc7bb2cd2e79b4628534fdf"}
Oct 30 22:14:30.072677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.072549    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"
Oct 30 22:14:30.072677 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.072564    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b26738402fa01b955e84ee049fb6afb1fd2a1c96bc7bb2cd2e79b4628534fdf"
Oct 30 22:14:30.236074 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.236011    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 22:14:30.348544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348506    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348554    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348631    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348706    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348745    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348839    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348886    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.348936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348887    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.348936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.348920    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.349208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo\") pod \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\" (UID: \"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb\") "
Oct 30 22:14:30.349272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349022    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.349272 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349037    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.349393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349116    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:30.349393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349254    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.349548 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349528    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.349596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349554    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.349596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349570    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.349596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349585    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.349753 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.349628    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.350957 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.350929    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.351076 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.350988    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:30.351494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.351469    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:30.351494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.351487    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq" pod "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" (UID: "e60f2c1e-2764-4e72-9ba2-24a821a9c2cb"). InnerVolumeSpecName "kube-api-access-hj7kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:30.423371 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.423339    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"]
Oct 30 22:14:30.451216 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.451186    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.451216 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.451215    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.451388 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.451229    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7kq\" (UniqueName: \"kubernetes.io/projected/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-kube-api-access-hj7kq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.451388 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.451244    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:30.824289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.824254    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"]
Oct 30 22:14:30.860849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:30.860814    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"]
Oct 30 22:14:31.099674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.090822    6695 generic.go:356] "Generic (PLEG): container finished" podID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerID="b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c" exitCode=0
Oct 30 22:14:31.099674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.091013    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" event={"ID":"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb","Type":"ContainerDied","Data":"b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c"}
Oct 30 22:14:31.099674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.091065    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz" event={"ID":"e60f2c1e-2764-4e72-9ba2-24a821a9c2cb","Type":"ContainerDied","Data":"e7d28b4a440ca520d52318a079716203eed05430831b1a78f3b5a64d7eb11188"}
Oct 30 22:14:31.099674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.091094    6695 scope.go:122] "RemoveContainer" containerID="b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c"
Oct 30 22:14:31.099674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.091237    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"
Oct 30 22:14:31.100342 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:31.100298    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:14:31.116978 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.116953    6695 scope.go:122] "RemoveContainer" containerID="4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"
Oct 30 22:14:31.126847 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.126820    6695 scope.go:122] "RemoveContainer" containerID="b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c"
Oct 30 22:14:31.127172 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:31.127150    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c\": container with ID starting with b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c not found: ID does not exist" containerID="b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c"
Oct 30 22:14:31.127256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.127179    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c"} err="failed to get container status \"b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c\": rpc error: code = NotFound desc = could not find container \"b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c\": container with ID starting with b71a8f6cb14925e3ac7de93bb5be56a506146fac756c4589d7508cd893d7247c not found: ID does not exist"
Oct 30 22:14:31.127256 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.127199    6695 scope.go:122] "RemoveContainer" containerID="4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"
Oct 30 22:14:31.127507 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:31.127487    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713\": container with ID starting with 4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713 not found: ID does not exist" containerID="4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"
Oct 30 22:14:31.127549 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.127520    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713"} err="failed to get container status \"4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713\": rpc error: code = NotFound desc = could not find container \"4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713\": container with ID starting with 4515625a83403eec8e773f33bce49381391f1e1ca64d353e44e753d552cb3713 not found: ID does not exist"
Oct 30 22:14:31.166395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.166359    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"]
Oct 30 22:14:31.170632 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.170585    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-bfabe1-openshift-default-57787d776f-j8pvz"]
Oct 30 22:14:31.214660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.214589    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" path="/var/lib/kubelet/pods/e60f2c1e-2764-4e72-9ba2-24a821a9c2cb/volumes"
Oct 30 22:14:31.668263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:31.668226    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"]
Oct 30 22:14:33.660045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.660021    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 22:14:33.675206 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.675175    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 22:14:33.785465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.785434    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms\") pod \"a0566572-3228-448b-8636-249effd583d4\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") "
Oct 30 22:14:33.785647 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.785567    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p\" (UniqueName: \"kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p\") pod \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") "
Oct 30 22:14:33.785699 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.785678    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume\") pod \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\" (UID: \"b4e0a1bc-3914-4fe0-9488-21cb77673acc\") "
Oct 30 22:14:33.785746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.785735    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume\") pod \"a0566572-3228-448b-8636-249effd583d4\" (UID: \"a0566572-3228-448b-8636-249effd583d4\") "
Oct 30 22:14:33.786077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.786051    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume" pod "b4e0a1bc-3914-4fe0-9488-21cb77673acc" (UID: "b4e0a1bc-3914-4fe0-9488-21cb77673acc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:33.786212 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.786156    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume" pod "a0566572-3228-448b-8636-249effd583d4" (UID: "a0566572-3228-448b-8636-249effd583d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:33.787694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.787671    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms" pod "a0566572-3228-448b-8636-249effd583d4" (UID: "a0566572-3228-448b-8636-249effd583d4"). InnerVolumeSpecName "kube-api-access-zs7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:33.788197 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.788171    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p" pod "b4e0a1bc-3914-4fe0-9488-21cb77673acc" (UID: "b4e0a1bc-3914-4fe0-9488-21cb77673acc"). InnerVolumeSpecName "kube-api-access-w5k4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:33.887187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.887085    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e0a1bc-3914-4fe0-9488-21cb77673acc-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:33.887187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.887129    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0566572-3228-448b-8636-249effd583d4-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:33.887187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.887144    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/a0566572-3228-448b-8636-249effd583d4-kube-api-access-zs7ms\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:33.887187 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:33.887158    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5k4p\" (UniqueName: \"kubernetes.io/projected/b4e0a1bc-3914-4fe0-9488-21cb77673acc-kube-api-access-w5k4p\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:34.116623 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.116573    6695 generic.go:356] "Generic (PLEG): container finished" podID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerID="abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854" exitCode=0
Oct 30 22:14:34.116831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.116637    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"
Oct 30 22:14:34.116831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.116713    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" event={"ID":"b4e0a1bc-3914-4fe0-9488-21cb77673acc","Type":"ContainerDied","Data":"abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854"}
Oct 30 22:14:34.116831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.116780    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb" event={"ID":"b4e0a1bc-3914-4fe0-9488-21cb77673acc","Type":"ContainerDied","Data":"c1c25b93c7cc4571c72dfb225d66200d43ad7931633354855e13387a3d2b536b"}
Oct 30 22:14:34.116831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.116813    6695 scope.go:122] "RemoveContainer" containerID="abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854"
Oct 30 22:14:34.118414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.118388    6695 generic.go:356] "Generic (PLEG): container finished" podID="a0566572-3228-448b-8636-249effd583d4" containerID="94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82" exitCode=0
Oct 30 22:14:34.118538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.118425    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"
Oct 30 22:14:34.118538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.118448    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" event={"ID":"a0566572-3228-448b-8636-249effd583d4","Type":"ContainerDied","Data":"94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82"}
Oct 30 22:14:34.118538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.118497    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j" event={"ID":"a0566572-3228-448b-8636-249effd583d4","Type":"ContainerDied","Data":"be42fdc42c21aea798fa8736dff1afe6278217d70c8f57768f50dfff337fa70f"}
Oct 30 22:14:34.133396 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.133337    6695 scope.go:122] "RemoveContainer" containerID="abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854"
Oct 30 22:14:34.133671 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:34.133651    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854\": container with ID starting with abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854 not found: ID does not exist" containerID="abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854"
Oct 30 22:14:34.133747 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.133679    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854"} err="failed to get container status \"abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854\": rpc error: code = NotFound desc = could not find container \"abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854\": container with ID starting with abbc59ed2616903cf3f7b53d4a4e2adbc708a90a33a0b53317ccdf248de13854 not found: ID does not exist"
Oct 30 22:14:34.133747 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.133695    6695 scope.go:122] "RemoveContainer" containerID="94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82"
Oct 30 22:14:34.143517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.143501    6695 scope.go:122] "RemoveContainer" containerID="94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82"
Oct 30 22:14:34.143810 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:34.143782    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82\": container with ID starting with 94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82 not found: ID does not exist" containerID="94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82"
Oct 30 22:14:34.143916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.143813    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82"} err="failed to get container status \"94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82\": rpc error: code = NotFound desc = could not find container \"94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82\": container with ID starting with 94b208f99a92fd42dc4cbff10f476ab8ac75768e1ba9547ac93dc175d503bb82 not found: ID does not exist"
Oct 30 22:14:34.184194 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.184162    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"]
Oct 30 22:14:34.195662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.195626    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ed0866-5bdf8769d7-jf72j"]
Oct 30 22:14:34.224701 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.224672    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"]
Oct 30 22:14:34.230962 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.230938    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ed0866-775fdc4cf5-xpwsb"]
Oct 30 22:14:34.248650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.248596    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:34.248899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.248876    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:34.359974 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.359943    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"]
Oct 30 22:14:34.363952 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.363920    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5494e8-59qdv"]
Oct 30 22:14:34.785960 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.785930    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 22:14:34.803265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.803243    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 22:14:34.899482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.899448    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh\" (UniqueName: \"kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh\") pod \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") "
Oct 30 22:14:34.899715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.899516    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc\" (UniqueName: \"kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc\") pod \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") "
Oct 30 22:14:34.899715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.899652    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume\") pod \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\" (UID: \"2e3307ce-1d41-45c8-b4a3-fd9bbc750813\") "
Oct 30 22:14:34.899715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.899679    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume\") pod \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\" (UID: \"9425b836-96bc-4f48-a6a1-5370bbc5ab0f\") "
Oct 30 22:14:34.900086 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.900053    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume" pod "2e3307ce-1d41-45c8-b4a3-fd9bbc750813" (UID: "2e3307ce-1d41-45c8-b4a3-fd9bbc750813"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:34.900289 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.900262    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume" pod "9425b836-96bc-4f48-a6a1-5370bbc5ab0f" (UID: "9425b836-96bc-4f48-a6a1-5370bbc5ab0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:34.901897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.901875    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc" pod "2e3307ce-1d41-45c8-b4a3-fd9bbc750813" (UID: "2e3307ce-1d41-45c8-b4a3-fd9bbc750813"). InnerVolumeSpecName "kube-api-access-s8tmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:34.902152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:34.902127    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh" pod "9425b836-96bc-4f48-a6a1-5370bbc5ab0f" (UID: "9425b836-96bc-4f48-a6a1-5370bbc5ab0f"). InnerVolumeSpecName "kube-api-access-krrvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:35.001660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.001626    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:35.001660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.001656    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:35.001660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.001665    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrvh\" (UniqueName: \"kubernetes.io/projected/9425b836-96bc-4f48-a6a1-5370bbc5ab0f-kube-api-access-krrvh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:35.001892 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.001677    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8tmc\" (UniqueName: \"kubernetes.io/projected/2e3307ce-1d41-45c8-b4a3-fd9bbc750813-kube-api-access-s8tmc\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:35.126931 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.126890    6695 generic.go:356] "Generic (PLEG): container finished" podID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerID="f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48" exitCode=0
Oct 30 22:14:35.127154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.126955    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"
Oct 30 22:14:35.127154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.126972    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" event={"ID":"2e3307ce-1d41-45c8-b4a3-fd9bbc750813","Type":"ContainerDied","Data":"f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48"}
Oct 30 22:14:35.127154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.127033    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh" event={"ID":"2e3307ce-1d41-45c8-b4a3-fd9bbc750813","Type":"ContainerDied","Data":"bbcfa3fe3b289ad64feb99436679aa24cc82828a6d8fa86979d32fba916f3f21"}
Oct 30 22:14:35.127154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.127066    6695 scope.go:122] "RemoveContainer" containerID="f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48"
Oct 30 22:14:35.129462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.129436    6695 generic.go:356] "Generic (PLEG): container finished" podID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerID="80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91" exitCode=0
Oct 30 22:14:35.129903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.129501    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"
Oct 30 22:14:35.129903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.129532    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" event={"ID":"9425b836-96bc-4f48-a6a1-5370bbc5ab0f","Type":"ContainerDied","Data":"80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91"}
Oct 30 22:14:35.129903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.129639    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr" event={"ID":"9425b836-96bc-4f48-a6a1-5370bbc5ab0f","Type":"ContainerDied","Data":"6981779eb80c6cbf5e136cba73aa70eeb4edd9ce246fedd50445a370631eb2f6"}
Oct 30 22:14:35.139210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.139186    6695 scope.go:122] "RemoveContainer" containerID="f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48"
Oct 30 22:14:35.140012 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:35.139672    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48\": container with ID starting with f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48 not found: ID does not exist" containerID="f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48"
Oct 30 22:14:35.140012 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.139712    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48"} err="failed to get container status \"f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48\": rpc error: code = NotFound desc = could not find container \"f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48\": container with ID starting with f6ae67c7d8ac8dd3d8d019a807540d1df307e84a6aaa115314fe6a4f7fafba48 not found: ID does not exist"
Oct 30 22:14:35.140012 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.139736    6695 scope.go:122] "RemoveContainer" containerID="80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91"
Oct 30 22:14:35.150888 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.150869    6695 scope.go:122] "RemoveContainer" containerID="80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91"
Oct 30 22:14:35.151186 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:35.151165    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91\": container with ID starting with 80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91 not found: ID does not exist" containerID="80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91"
Oct 30 22:14:35.151264 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.151193    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91"} err="failed to get container status \"80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91\": rpc error: code = NotFound desc = could not find container \"80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91\": container with ID starting with 80bd64d813c837470d12fd6d7839e48e68bee34c8b9c8a6c5b193ab9ab10ca91 not found: ID does not exist"
Oct 30 22:14:35.196063 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.196028    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"]
Oct 30 22:14:35.213158 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.213124    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c86752-4733-41d6-8473-eb703c91f331" path="/var/lib/kubelet/pods/64c86752-4733-41d6-8473-eb703c91f331/volumes"
Oct 30 22:14:35.213533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.213520    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0566572-3228-448b-8636-249effd583d4" path="/var/lib/kubelet/pods/a0566572-3228-448b-8636-249effd583d4/volumes"
Oct 30 22:14:35.213897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.213884    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" path="/var/lib/kubelet/pods/b4e0a1bc-3914-4fe0-9488-21cb77673acc/volumes"
Oct 30 22:14:35.214173 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.214161    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9722ac-695c6b8cfc-8tlmr"]
Oct 30 22:14:35.235708 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.235666    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"]
Oct 30 22:14:35.237143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.237122    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9722ac-77485758ff-4frhh"]
Oct 30 22:14:35.824808 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.824735    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" containerName="payload-processing" containerID="cri-o://1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf" gracePeriod=25
Oct 30 22:14:35.861411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:35.861343    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerName="payload-pre-processing" containerID="cri-o://074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61" gracePeriod=25
Oct 30 22:14:36.152789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:36.152709    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"]
Oct 30 22:14:36.182170 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:36.182136    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"]
Oct 30 22:14:36.668779 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:36.668731    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerName="payload-pre-processing" containerID="cri-o://601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db" gracePeriod=25
Oct 30 22:14:37.213254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:37.213216    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" path="/var/lib/kubelet/pods/2e3307ce-1d41-45c8-b4a3-fd9bbc750813/volumes"
Oct 30 22:14:37.213715 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:37.213698    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" path="/var/lib/kubelet/pods/9425b836-96bc-4f48-a6a1-5370bbc5ab0f/volumes"
Oct 30 22:14:37.298806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:37.298763    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:37.298806 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:37.298776    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:38.200455 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:38.200412    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:38.200673 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:38.200411    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:40.842169 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.842139    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 22:14:40.888593 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.887375    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 22:14:40.970400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.970368    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume\") pod \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") "
Oct 30 22:14:40.970581 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.970425    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq\" (UniqueName: \"kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq\") pod \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\" (UID: \"8d4933a4-2dc2-41f1-978c-56b2b30694b2\") "
Oct 30 22:14:40.970581 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.970496    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh\" (UniqueName: \"kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh\") pod \"a25652f6-e0f8-4960-8d1b-59b5686054c6\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") "
Oct 30 22:14:40.970801 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.970579    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume\") pod \"a25652f6-e0f8-4960-8d1b-59b5686054c6\" (UID: \"a25652f6-e0f8-4960-8d1b-59b5686054c6\") "
Oct 30 22:14:40.970886 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.970847    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume" pod "8d4933a4-2dc2-41f1-978c-56b2b30694b2" (UID: "8d4933a4-2dc2-41f1-978c-56b2b30694b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:40.971011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.970983    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume" pod "a25652f6-e0f8-4960-8d1b-59b5686054c6" (UID: "a25652f6-e0f8-4960-8d1b-59b5686054c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:40.971219 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.971200    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4933a4-2dc2-41f1-978c-56b2b30694b2-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:40.971269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.971226    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25652f6-e0f8-4960-8d1b-59b5686054c6-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:40.972596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.972575    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh" pod "a25652f6-e0f8-4960-8d1b-59b5686054c6" (UID: "a25652f6-e0f8-4960-8d1b-59b5686054c6"). InnerVolumeSpecName "kube-api-access-lxghh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:40.972751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:40.972732    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq" pod "8d4933a4-2dc2-41f1-978c-56b2b30694b2" (UID: "8d4933a4-2dc2-41f1-978c-56b2b30694b2"). InnerVolumeSpecName "kube-api-access-h8vzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:41.072441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.072365    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8vzq\" (UniqueName: \"kubernetes.io/projected/8d4933a4-2dc2-41f1-978c-56b2b30694b2-kube-api-access-h8vzq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:41.072441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.072394    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxghh\" (UniqueName: \"kubernetes.io/projected/a25652f6-e0f8-4960-8d1b-59b5686054c6-kube-api-access-lxghh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:41.153361 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.153314    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerName="payload-processing" containerID="cri-o://e2e68297cbc3556a5e294773124501153ff096c21bf67f90b5ab0f3400693b03" gracePeriod=25
Oct 30 22:14:41.182668 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.182451    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerName="payload-pre-processing" containerID="cri-o://0de39b3c68e20a4c098a17370f36b86bac13833719ca71ed98e2275409d5dd41" gracePeriod=25
Oct 30 22:14:41.183249 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.183089    6695 generic.go:356] "Generic (PLEG): container finished" podID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerID="37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119" exitCode=0
Oct 30 22:14:41.183249 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.183160    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" event={"ID":"a25652f6-e0f8-4960-8d1b-59b5686054c6","Type":"ContainerDied","Data":"37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119"}
Oct 30 22:14:41.183481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.183171    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"
Oct 30 22:14:41.183481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.183280    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8" event={"ID":"a25652f6-e0f8-4960-8d1b-59b5686054c6","Type":"ContainerDied","Data":"129addf1a7b1ac3beb2d42dc279d31a8bdc05a8b7d1f47766f4f2bc8f1c2e484"}
Oct 30 22:14:41.183481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.183325    6695 scope.go:122] "RemoveContainer" containerID="37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119"
Oct 30 22:14:41.185691 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.185663    6695 generic.go:356] "Generic (PLEG): container finished" podID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerID="657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960" exitCode=0
Oct 30 22:14:41.185816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.185725    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"
Oct 30 22:14:41.185816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.185743    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" event={"ID":"8d4933a4-2dc2-41f1-978c-56b2b30694b2","Type":"ContainerDied","Data":"657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960"}
Oct 30 22:14:41.185941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.185817    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh" event={"ID":"8d4933a4-2dc2-41f1-978c-56b2b30694b2","Type":"ContainerDied","Data":"ce008a1ef7fff67d71ab23f944772372641a820d4a5bfb84d96cbb20ebf520c1"}
Oct 30 22:14:41.194629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.194578    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"]
Oct 30 22:14:41.195210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.194856    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy" containerID="cri-o://129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912" gracePeriod=30
Oct 30 22:14:41.209095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.209076    6695 scope.go:122] "RemoveContainer" containerID="37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119"
Oct 30 22:14:41.209454 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:41.209430    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119\": container with ID starting with 37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119 not found: ID does not exist" containerID="37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119"
Oct 30 22:14:41.209558 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.209463    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119"} err="failed to get container status \"37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119\": rpc error: code = NotFound desc = could not find container \"37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119\": container with ID starting with 37dcca2009908c4d3d608c72a0c1a7bf6a440505d61af02e920f3c91bbb52119 not found: ID does not exist"
Oct 30 22:14:41.209558 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.209480    6695 scope.go:122] "RemoveContainer" containerID="657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960"
Oct 30 22:14:41.219914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.219896    6695 scope.go:122] "RemoveContainer" containerID="657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960"
Oct 30 22:14:41.220181 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:41.220162    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960\": container with ID starting with 657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960 not found: ID does not exist" containerID="657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960"
Oct 30 22:14:41.220235 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.220189    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960"} err="failed to get container status \"657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960\": rpc error: code = NotFound desc = could not find container \"657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960\": container with ID starting with 657163c1465015c8c3eaccdc5b0ee57b5ef0b83f3ceab1bd07e6b86ceb3e6960 not found: ID does not exist"
Oct 30 22:14:41.302540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.302505    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"]
Oct 30 22:14:41.306095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.306063    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-bfabe1-bf7fdd859-jdcm8"]
Oct 30 22:14:41.334090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.334003    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"]
Oct 30 22:14:41.337374 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.337346    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-bfabe1-6d4df6bc79-5s7qh"]
Oct 30 22:14:41.765135 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765059    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"]
Oct 30 22:14:41.765538 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765526    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerName="payload-pre-processing"
Oct 30 22:14:41.765590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765540    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerName="payload-pre-processing"
Oct 30 22:14:41.765590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765564    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy"
Oct 30 22:14:41.765590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765569    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy"
Oct 30 22:14:41.765590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765577    6695 cpu_manager.go:414] "containerMap: removing container" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerName="payload-pre-processing"
Oct 30 22:14:41.765590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765582    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerName="payload-pre-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765594    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerName="payload-pre-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765626    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerName="payload-pre-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765639    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765648    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765658    6695 cpu_manager.go:414] "containerMap: removing container" podUID="a0566572-3228-448b-8636-249effd583d4" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765663    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0566572-3228-448b-8636-249effd583d4" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765674    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3443d6a4-2241-48fe-ad9d-e3910403d27b" containerName="revoke-keys"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765679    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3443d6a4-2241-48fe-ad9d-e3910403d27b" containerName="revoke-keys"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765694    6695 cpu_manager.go:414] "containerMap: removing container" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765701    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765777    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3443d6a4-2241-48fe-ad9d-e3910403d27b" containerName="revoke-keys"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765787    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765793    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="9425b836-96bc-4f48-a6a1-5370bbc5ab0f" containerName="payload-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765801    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3307ce-1d41-45c8-b4a3-fd9bbc750813" containerName="payload-pre-processing"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765808    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="406109b5-5975-4c91-95e7-739fa653abbb" containerName="istio-proxy"
Oct 30 22:14:41.765810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765816    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" containerName="payload-pre-processing"
Oct 30 22:14:41.766393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765824    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e0a1bc-3914-4fe0-9488-21cb77673acc" containerName="payload-pre-processing"
Oct 30 22:14:41.766393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765830    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" containerName="payload-processing"
Oct 30 22:14:41.766393 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.765836    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0566572-3228-448b-8636-249effd583d4" containerName="payload-processing"
Oct 30 22:14:41.770788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.770769    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.788018 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.787990    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"]
Oct 30 22:14:41.796111 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.796088    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:14:41.796251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.796115    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:14:41.866908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.866877    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:41.866908 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.866911    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:14:41.883910 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.883887    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt6ll\" (UniqueName: \"kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.884031 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.883932    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.884146 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.884128    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.985781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.985749    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.985977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.985797    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt6ll\" (UniqueName: \"kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.985977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.985833    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.986446 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.986408    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:41.988160 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:41.988139    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:42.005296 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:42.005269    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt6ll\" (UniqueName: \"kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:42.083967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:42.083932    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:42.392662 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:42.392562    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"]
Oct 30 22:14:42.396142 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:42.396110    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c40ced3_974b_4945_ad12_87dd51f84222.slice/crio-b7946cfa26c48a56e32b4a1908b63eaf25601ff1f5d4fbcee9868e8f0b2bae03 WatchSource:0}: Error finding container b7946cfa26c48a56e32b4a1908b63eaf25601ff1f5d4fbcee9868e8f0b2bae03: Status 404 returned error can't find the container with id b7946cfa26c48a56e32b4a1908b63eaf25601ff1f5d4fbcee9868e8f0b2bae03
Oct 30 22:14:43.203297 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.203261    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z" event={"ID":"7c40ced3-974b-4945-ad12-87dd51f84222","Type":"ContainerStarted","Data":"bed689f8f8d6ff48644d5a60180882fa9d8cd69e18e973f385e50fff8f27d96d"}
Oct 30 22:14:43.203724 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.203315    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z" event={"ID":"7c40ced3-974b-4945-ad12-87dd51f84222","Type":"ContainerStarted","Data":"b7946cfa26c48a56e32b4a1908b63eaf25601ff1f5d4fbcee9868e8f0b2bae03"}
Oct 30 22:14:43.213740 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.213707    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4933a4-2dc2-41f1-978c-56b2b30694b2" path="/var/lib/kubelet/pods/8d4933a4-2dc2-41f1-978c-56b2b30694b2/volumes"
Oct 30 22:14:43.214224 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.214201    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25652f6-e0f8-4960-8d1b-59b5686054c6" path="/var/lib/kubelet/pods/a25652f6-e0f8-4960-8d1b-59b5686054c6/volumes"
Oct 30 22:14:43.259273 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.259200    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z" podStartSLOduration=2.259178988 podStartE2EDuration="2.259178988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:43.246865386 +0000 UTC m=+3866.169323686" watchObservedRunningTime="2026-10-30 22:14:43.259178988 +0000 UTC m=+3866.181637288"
Oct 30 22:14:43.308921 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.308890    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"]
Oct 30 22:14:43.309454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.309436    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy"
Oct 30 22:14:43.309454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.309452    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy"
Oct 30 22:14:43.309688 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.309665    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60f2c1e-2764-4e72-9ba2-24a821a9c2cb" containerName="istio-proxy"
Oct 30 22:14:43.314266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.314245    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"]
Oct 30 22:14:43.314407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.314386    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.318432 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.318410    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.337543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.337507    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"]
Oct 30 22:14:43.343085 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.343034    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"]
Oct 30 22:14:43.402578 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.402544    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.402748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.402585    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.402828 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.402746    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkml\" (UniqueName: \"kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.402893 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.402834    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.403120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.402981    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.403240 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.403133    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4lp\" (UniqueName: \"kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.504274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.504159    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.504274 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.504233    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkml\" (UniqueName: \"kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.504528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.504347    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.504528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.504490    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.504528 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.504522    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4lp\" (UniqueName: \"kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.504737 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.504634    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.506027 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.506000    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.506508 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.506473    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.509204 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.509178    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.509562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.509537    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.530089 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.530049    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkml\" (UniqueName: \"kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml\") pod \"maas-api-revoke-sub-d13eb1d2-t8gv9\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:43.531750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.531502    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4lp\" (UniqueName: \"kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp\") pod \"maas-api-revoke-sub-3970c8af-x2chp\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.640236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.640203    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"]
Oct 30 22:14:43.662657 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.662615    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-e06b90-zftxz"]
Oct 30 22:14:43.702199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.702160    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:43.710434 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:43.710414    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:44.011249 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:44.011175    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"]
Oct 30 22:14:44.015797 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:44.015767    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717813ec_d5ff_4cac_9cfe_39a4aedf14b1.slice/crio-0f57a9a1074832317ccd86db2cf95b66f23bae63c91d2a47f8b2398f6482cb26 WatchSource:0}: Error finding container 0f57a9a1074832317ccd86db2cf95b66f23bae63c91d2a47f8b2398f6482cb26: Status 404 returned error can't find the container with id 0f57a9a1074832317ccd86db2cf95b66f23bae63c91d2a47f8b2398f6482cb26
Oct 30 22:14:44.047023 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:44.045170    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"]
Oct 30 22:14:44.060809 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:44.060780    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0cfa60_a3fd_47c6_9b8b_532f5be42f8c.slice/crio-5e48b0ef920ac895081056f50ab6ef6c647a58e99dcf4527fc54ebc8cd494dc4 WatchSource:0}: Error finding container 5e48b0ef920ac895081056f50ab6ef6c647a58e99dcf4527fc54ebc8cd494dc4: Status 404 returned error can't find the container with id 5e48b0ef920ac895081056f50ab6ef6c647a58e99dcf4527fc54ebc8cd494dc4
Oct 30 22:14:44.214592 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:44.214545    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9" event={"ID":"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c","Type":"ContainerStarted","Data":"5e48b0ef920ac895081056f50ab6ef6c647a58e99dcf4527fc54ebc8cd494dc4"}
Oct 30 22:14:44.216534 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:44.216499    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp" event={"ID":"717813ec-d5ff-4cac-9cfe-39a4aedf14b1","Type":"ContainerStarted","Data":"0f57a9a1074832317ccd86db2cf95b66f23bae63c91d2a47f8b2398f6482cb26"}
Oct 30 22:14:44.219705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:44.219676    6695 generic.go:356] "Generic (PLEG): container finished" podID="7c40ced3-974b-4945-ad12-87dd51f84222" containerID="bed689f8f8d6ff48644d5a60180882fa9d8cd69e18e973f385e50fff8f27d96d" exitCode=0
Oct 30 22:14:44.219986 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:44.219964    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z" event={"ID":"7c40ced3-974b-4945-ad12-87dd51f84222","Type":"ContainerDied","Data":"bed689f8f8d6ff48644d5a60180882fa9d8cd69e18e973f385e50fff8f27d96d"}
Oct 30 22:14:45.213283 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.213204    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f92ebe-680e-4226-90b3-039d49b79887" path="/var/lib/kubelet/pods/71f92ebe-680e-4226-90b3-039d49b79887/volumes"
Oct 30 22:14:45.228208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.228175    6695 generic.go:356] "Generic (PLEG): container finished" podID="5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" containerID="fb7422af3abf5982f9ba853658c177eefea85f481965d92d789cc7dd75f0902a" exitCode=0
Oct 30 22:14:45.228767 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.228265    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9" event={"ID":"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c","Type":"ContainerDied","Data":"fb7422af3abf5982f9ba853658c177eefea85f481965d92d789cc7dd75f0902a"}
Oct 30 22:14:45.230094 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.230064    6695 generic.go:356] "Generic (PLEG): container finished" podID="717813ec-d5ff-4cac-9cfe-39a4aedf14b1" containerID="21505a95f5d55aac017333e7954bf62e88ae4dc456734c7a0147045a9c5c2520" exitCode=0
Oct 30 22:14:45.230210 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.230142    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp" event={"ID":"717813ec-d5ff-4cac-9cfe-39a4aedf14b1","Type":"ContainerDied","Data":"21505a95f5d55aac017333e7954bf62e88ae4dc456734c7a0147045a9c5c2520"}
Oct 30 22:14:45.379899 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.379872    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:45.444457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.444412    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main" containerID="cri-o://048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34" gracePeriod=45
Oct 30 22:14:45.528404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.528373    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll\" (UniqueName: \"kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll\") pod \"7c40ced3-974b-4945-ad12-87dd51f84222\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") "
Oct 30 22:14:45.528594 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.528490    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca\") pod \"7c40ced3-974b-4945-ad12-87dd51f84222\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") "
Oct 30 22:14:45.528594 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.528527    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca\") pod \"7c40ced3-974b-4945-ad12-87dd51f84222\" (UID: \"7c40ced3-974b-4945-ad12-87dd51f84222\") "
Oct 30 22:14:45.528880 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.528846    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca" pod "7c40ced3-974b-4945-ad12-87dd51f84222" (UID: "7c40ced3-974b-4945-ad12-87dd51f84222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:45.529052 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.529018    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:45.529179 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.529080    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.530741 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.530714    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll" pod "7c40ced3-974b-4945-ad12-87dd51f84222" (UID: "7c40ced3-974b-4945-ad12-87dd51f84222"). InnerVolumeSpecName "kube-api-access-wt6ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:45.530856 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.530801    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca" pod "7c40ced3-974b-4945-ad12-87dd51f84222" (UID: "7c40ced3-974b-4945-ad12-87dd51f84222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:45.639914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.639869    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c40ced3-974b-4945-ad12-87dd51f84222-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.639914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.639916    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt6ll\" (UniqueName: \"kubernetes.io/projected/7c40ced3-974b-4945-ad12-87dd51f84222-kube-api-access-wt6ll\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.702017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.701990    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 22:14:45.842230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842144    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home\") pod \"4616cf15-31ad-470b-823b-e99e2143d885\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") "
Oct 30 22:14:45.842230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842231    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir\") pod \"4616cf15-31ad-470b-823b-e99e2143d885\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") "
Oct 30 22:14:45.842491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842324    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs\" (UniqueName: \"kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs\") pod \"4616cf15-31ad-470b-823b-e99e2143d885\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") "
Oct 30 22:14:45.842491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842361    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht\" (UniqueName: \"kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht\") pod \"4616cf15-31ad-470b-823b-e99e2143d885\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") "
Oct 30 22:14:45.842491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842408    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache\") pod \"4616cf15-31ad-470b-823b-e99e2143d885\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") "
Oct 30 22:14:45.842734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842493    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm\") pod \"4616cf15-31ad-470b-823b-e99e2143d885\" (UID: \"4616cf15-31ad-470b-823b-e99e2143d885\") "
Oct 30 22:14:45.842734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842470    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home" pod "4616cf15-31ad-470b-823b-e99e2143d885" (UID: "4616cf15-31ad-470b-823b-e99e2143d885"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:45.842734 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842539    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir" pod "4616cf15-31ad-470b-823b-e99e2143d885" (UID: "4616cf15-31ad-470b-823b-e99e2143d885"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:45.842914 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.842767    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache" pod "4616cf15-31ad-470b-823b-e99e2143d885" (UID: "4616cf15-31ad-470b-823b-e99e2143d885"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:45.843389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.843361    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.843389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.843393    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.843389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.843406    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.845135 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.845094    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs" pod "4616cf15-31ad-470b-823b-e99e2143d885" (UID: "4616cf15-31ad-470b-823b-e99e2143d885"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:45.845254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.845183    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm" pod "4616cf15-31ad-470b-823b-e99e2143d885" (UID: "4616cf15-31ad-470b-823b-e99e2143d885"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:45.845254 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.845197    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht" pod "4616cf15-31ad-470b-823b-e99e2143d885" (UID: "4616cf15-31ad-470b-823b-e99e2143d885"). InnerVolumeSpecName "kube-api-access-f2mht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:45.944413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.944374    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4616cf15-31ad-470b-823b-e99e2143d885-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.944413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.944411    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mht\" (UniqueName: \"kubernetes.io/projected/4616cf15-31ad-470b-823b-e99e2143d885-kube-api-access-f2mht\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:45.944413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:45.944420    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4616cf15-31ad-470b-823b-e99e2143d885-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.237405 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.237377    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"
Oct 30 22:14:46.237791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.237403    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z" event={"ID":"7c40ced3-974b-4945-ad12-87dd51f84222","Type":"ContainerDied","Data":"b7946cfa26c48a56e32b4a1908b63eaf25601ff1f5d4fbcee9868e8f0b2bae03"}
Oct 30 22:14:46.237791 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.237459    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7946cfa26c48a56e32b4a1908b63eaf25601ff1f5d4fbcee9868e8f0b2bae03"
Oct 30 22:14:46.238945 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.238920    6695 generic.go:356] "Generic (PLEG): container finished" podID="4616cf15-31ad-470b-823b-e99e2143d885" containerID="048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34" exitCode=0
Oct 30 22:14:46.239059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.238970    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"
Oct 30 22:14:46.239059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.238998    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" event={"ID":"4616cf15-31ad-470b-823b-e99e2143d885","Type":"ContainerDied","Data":"048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34"}
Oct 30 22:14:46.239059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.239040    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86" event={"ID":"4616cf15-31ad-470b-823b-e99e2143d885","Type":"ContainerDied","Data":"85c0fc0bee53606614073839fc14656627cf3d858e7802bad8ec51d36c178768"}
Oct 30 22:14:46.239059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.239058    6695 scope.go:122] "RemoveContainer" containerID="048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34"
Oct 30 22:14:46.323804 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.323777    6695 scope.go:122] "RemoveContainer" containerID="048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34"
Oct 30 22:14:46.330160 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.330131    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"]
Oct 30 22:14:46.332955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.332929    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b31518/premium-w1-b31518-kserve-b76879949-qmb86"]
Oct 30 22:14:46.333442 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:46.333410    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34\": container with ID starting with 048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34 not found: ID does not exist" containerID="048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34"
Oct 30 22:14:46.333556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.333454    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34"} err="failed to get container status \"048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34\": rpc error: code = NotFound desc = could not find container \"048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34\": container with ID starting with 048ca00531178cc07b531700b7059d33e2984dd3fbbe254a37d6bf43cf83aa34 not found: ID does not exist"
Oct 30 22:14:46.408033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.407720    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:46.410163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.410136    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:46.497818 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.497785    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 22:14:46.549939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.549907    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca\") pod \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") "
Oct 30 22:14:46.550154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.549984    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca\") pod \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") "
Oct 30 22:14:46.550241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550146    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca\") pod \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") "
Oct 30 22:14:46.550241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550215    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp\" (UniqueName: \"kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp\") pod \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\" (UID: \"717813ec-d5ff-4cac-9cfe-39a4aedf14b1\") "
Oct 30 22:14:46.550363 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550296    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca" pod "717813ec-d5ff-4cac-9cfe-39a4aedf14b1" (UID: "717813ec-d5ff-4cac-9cfe-39a4aedf14b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:46.550363 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550313    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca\") pod \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") "
Oct 30 22:14:46.550493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550373    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml\" (UniqueName: \"kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml\") pod \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\" (UID: \"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c\") "
Oct 30 22:14:46.550689 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550664    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca" pod "5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" (UID: "5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:46.550906 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550878    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.550906 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.550908    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.552225 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.552200    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca" pod "5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" (UID: "5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:46.552589 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.552561    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml" pod "5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" (UID: "5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c"). InnerVolumeSpecName "kube-api-access-4lkml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:46.552717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.552666    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca" pod "717813ec-d5ff-4cac-9cfe-39a4aedf14b1" (UID: "717813ec-d5ff-4cac-9cfe-39a4aedf14b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:46.552903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.552884    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp" pod "717813ec-d5ff-4cac-9cfe-39a4aedf14b1" (UID: "717813ec-d5ff-4cac-9cfe-39a4aedf14b1"). InnerVolumeSpecName "kube-api-access-zq4lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:46.651719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651685    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.651902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651759    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.651902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651791    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.651902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651815    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.651902 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651865    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.652137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651937    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.652137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651963    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.652137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.651980    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:46.652137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652001    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.652137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652098    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:46.652423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652266    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token\") pod \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\" (UID: \"e4c4c2b4-4a4b-41d6-aa81-66f0da709615\") "
Oct 30 22:14:46.652423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652377    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:46.652423 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652390    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:46.652586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652442    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:46.652765 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652747    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.652845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652771    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.652845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652785    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.652845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652800    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4lp\" (UniqueName: \"kubernetes.io/projected/717813ec-d5ff-4cac-9cfe-39a4aedf14b1-kube-api-access-zq4lp\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.652845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652815    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.652845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652828    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.652845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652841    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.653080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652854    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lkml\" (UniqueName: \"kubernetes.io/projected/5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c-kube-api-access-4lkml\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.653080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.652879    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.654081 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.654061    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:46.654134 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.654095    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "kube-api-access-8vxhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:46.654344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.654330    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:46.654424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.654409    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token" pod "e4c4c2b4-4a4b-41d6-aa81-66f0da709615" (UID: "e4c4c2b4-4a4b-41d6-aa81-66f0da709615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:46.753595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.753519    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxhq\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-kube-api-access-8vxhq\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.753595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.753550    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.753595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.753563    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:46.753595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:46.753574    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c4c2b4-4a4b-41d6-aa81-66f0da709615-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:47.213859 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.213820    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4616cf15-31ad-470b-823b-e99e2143d885" path="/var/lib/kubelet/pods/4616cf15-31ad-470b-823b-e99e2143d885/volumes"
Oct 30 22:14:47.247248 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.247223    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9"
Oct 30 22:14:47.247248 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.247240    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13eb1d2-t8gv9" event={"ID":"5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c","Type":"ContainerDied","Data":"5e48b0ef920ac895081056f50ab6ef6c647a58e99dcf4527fc54ebc8cd494dc4"}
Oct 30 22:14:47.247660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.247287    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e48b0ef920ac895081056f50ab6ef6c647a58e99dcf4527fc54ebc8cd494dc4"
Oct 30 22:14:47.248955 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.248934    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp"
Oct 30 22:14:47.249095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.248951    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3970c8af-x2chp" event={"ID":"717813ec-d5ff-4cac-9cfe-39a4aedf14b1","Type":"ContainerDied","Data":"0f57a9a1074832317ccd86db2cf95b66f23bae63c91d2a47f8b2398f6482cb26"}
Oct 30 22:14:47.249095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.248999    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f57a9a1074832317ccd86db2cf95b66f23bae63c91d2a47f8b2398f6482cb26"
Oct 30 22:14:47.253309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.253285    6695 generic.go:356] "Generic (PLEG): container finished" podID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerID="129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912" exitCode=0
Oct 30 22:14:47.253433 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.253327    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"
Oct 30 22:14:47.253433 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.253367    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" event={"ID":"e4c4c2b4-4a4b-41d6-aa81-66f0da709615","Type":"ContainerDied","Data":"129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912"}
Oct 30 22:14:47.253433 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.253415    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82" event={"ID":"e4c4c2b4-4a4b-41d6-aa81-66f0da709615","Type":"ContainerDied","Data":"fc984bb19ec75e6f13c4cfd7a70f1bec8fa96340a99208fcfc60c7ca2baff310"}
Oct 30 22:14:47.253570 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.253443    6695 scope.go:122] "RemoveContainer" containerID="129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912"
Oct 30 22:14:47.263973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.263948    6695 scope.go:122] "RemoveContainer" containerID="75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"
Oct 30 22:14:47.274445 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.274421    6695 scope.go:122] "RemoveContainer" containerID="129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912"
Oct 30 22:14:47.274758 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:47.274730    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912\": container with ID starting with 129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912 not found: ID does not exist" containerID="129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912"
Oct 30 22:14:47.274838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.274759    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912"} err="failed to get container status \"129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912\": rpc error: code = NotFound desc = could not find container \"129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912\": container with ID starting with 129c529dfd7502ed2c3c888d140e6603d3cee8a7c30b6f8f152c59d0f76f5912 not found: ID does not exist"
Oct 30 22:14:47.274838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.274778    6695 scope.go:122] "RemoveContainer" containerID="75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"
Oct 30 22:14:47.275061 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:47.275039    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35\": container with ID starting with 75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35 not found: ID does not exist" containerID="75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"
Oct 30 22:14:47.275130 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.275071    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35"} err="failed to get container status \"75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35\": rpc error: code = NotFound desc = could not find container \"75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35\": container with ID starting with 75f022404dc9b171569a62f6f0db6d866792df2a4752130aa430314f79a2ec35 not found: ID does not exist"
Oct 30 22:14:47.335118 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.335082    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"]
Oct 30 22:14:47.347463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.347415    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-2sb82"]
Oct 30 22:14:47.505704 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.505617    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:47.505863 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:47.505623    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:48.199770 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:48.199726    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:48.199931 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:48.199744    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:49.215040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:49.215001    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" path="/var/lib/kubelet/pods/e4c4c2b4-4a4b-41d6-aa81-66f0da709615/volumes"
Oct 30 22:14:52.314864 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:52.314819    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"]
Oct 30 22:14:53.483717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.483677    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"]
Oct 30 22:14:53.484193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484139    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy"
Oct 30 22:14:53.484193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484150    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy"
Oct 30 22:14:53.484193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484183    6695 cpu_manager.go:414] "containerMap: removing container" podUID="717813ec-d5ff-4cac-9cfe-39a4aedf14b1" containerName="revoke-keys"
Oct 30 22:14:53.484193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484189    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717813ec-d5ff-4cac-9cfe-39a4aedf14b1" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484198    6695 cpu_manager.go:414] "containerMap: removing container" podUID="7c40ced3-974b-4945-ad12-87dd51f84222" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484204    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c40ced3-974b-4945-ad12-87dd51f84222" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484217    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484221    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484230    6695 cpu_manager.go:414] "containerMap: removing container" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484234    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484295    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484304    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="717813ec-d5ff-4cac-9cfe-39a4aedf14b1" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484313    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c40ced3-974b-4945-ad12-87dd51f84222" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484319    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c" containerName="revoke-keys"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484325    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="4616cf15-31ad-470b-823b-e99e2143d885" containerName="main"
Oct 30 22:14:53.484453 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.484333    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy"
Oct 30 22:14:53.489119 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.489101    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.516132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.516099    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:14:53.516638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.516129    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:53.516824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.516501    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:14:53.516968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.516564    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:14:53.519694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.519671    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"]
Oct 30 22:14:53.521456 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.521426    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.521543 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.521530    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p6q\" (UniqueName: \"kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.521717 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.521692    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.622587 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.622553    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p6q\" (UniqueName: \"kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.622814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.622663    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.622814 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.622745    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.623332 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.623309    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.625003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.624979    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.642489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.642456    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p6q\" (UniqueName: \"kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:53.802463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:53.802425    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:54.104781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:54.104698    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"]
Oct 30 22:14:54.108448 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:14:54.108412    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb229af07_92f7_424e_89a1_25c699430029.slice/crio-91d9945771c5835d999695dfdab3e4261ddf0a4be35dd5bf2429bc500d91441b WatchSource:0}: Error finding container 91d9945771c5835d999695dfdab3e4261ddf0a4be35dd5bf2429bc500d91441b: Status 404 returned error can't find the container with id 91d9945771c5835d999695dfdab3e4261ddf0a4be35dd5bf2429bc500d91441b
Oct 30 22:14:54.303477 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:54.303436    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz" event={"ID":"b229af07-92f7-424e-89a1-25c699430029","Type":"ContainerStarted","Data":"91d9945771c5835d999695dfdab3e4261ddf0a4be35dd5bf2429bc500d91441b"}
Oct 30 22:14:55.311339 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:55.311308    6695 generic.go:356] "Generic (PLEG): container finished" podID="b229af07-92f7-424e-89a1-25c699430029" containerID="83d2205560e6a0b1d2bc33a1ddc9dcc4f8ccf60cab66a7aa1ebdc625da104f28" exitCode=0
Oct 30 22:14:55.311769 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:55.311395    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz" event={"ID":"b229af07-92f7-424e-89a1-25c699430029","Type":"ContainerDied","Data":"83d2205560e6a0b1d2bc33a1ddc9dcc4f8ccf60cab66a7aa1ebdc625da104f28"}
Oct 30 22:14:55.529239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:55.529199    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:56.120465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.120435    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 22:14:56.150090 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.150061    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 22:14:56.151121 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.151102    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz\" (UniqueName: \"kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz\") pod \"1276deaf-a385-4f4d-887c-8214e584cf68\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") "
Oct 30 22:14:56.151193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.151138    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume\") pod \"1276deaf-a385-4f4d-887c-8214e584cf68\" (UID: \"1276deaf-a385-4f4d-887c-8214e584cf68\") "
Oct 30 22:14:56.151595 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.151575    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume" pod "1276deaf-a385-4f4d-887c-8214e584cf68" (UID: "1276deaf-a385-4f4d-887c-8214e584cf68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.153133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.153106    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz" pod "1276deaf-a385-4f4d-887c-8214e584cf68" (UID: "1276deaf-a385-4f4d-887c-8214e584cf68"). InnerVolumeSpecName "kube-api-access-xrzxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:56.251957 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.251920    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c\" (UniqueName: \"kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c\") pod \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") "
Oct 30 22:14:56.252158 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.252067    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume\") pod \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\" (UID: \"0eb24b4a-0af8-491b-b6bb-7c5fe3353348\") "
Oct 30 22:14:56.252369 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.252352    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzxz\" (UniqueName: \"kubernetes.io/projected/1276deaf-a385-4f4d-887c-8214e584cf68-kube-api-access-xrzxz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.252369 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.252367    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1276deaf-a385-4f4d-887c-8214e584cf68-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.252482 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.252399    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume" pod "0eb24b4a-0af8-491b-b6bb-7c5fe3353348" (UID: "0eb24b4a-0af8-491b-b6bb-7c5fe3353348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.254054 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.254036    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c" pod "0eb24b4a-0af8-491b-b6bb-7c5fe3353348" (UID: "0eb24b4a-0af8-491b-b6bb-7c5fe3353348"). InnerVolumeSpecName "kube-api-access-vgx6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:56.319307 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.319271    6695 generic.go:356] "Generic (PLEG): container finished" podID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerID="074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61" exitCode=0
Oct 30 22:14:56.319781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.319316    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"
Oct 30 22:14:56.319781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.319355    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" event={"ID":"0eb24b4a-0af8-491b-b6bb-7c5fe3353348","Type":"ContainerDied","Data":"074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61"}
Oct 30 22:14:56.319781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.319409    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7" event={"ID":"0eb24b4a-0af8-491b-b6bb-7c5fe3353348","Type":"ContainerDied","Data":"cab0dffc3316868599e82b92746eee5b4c56bd3e4b68df861f08c176bbf5ed10"}
Oct 30 22:14:56.319781 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.319439    6695 scope.go:122] "RemoveContainer" containerID="074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61"
Oct 30 22:14:56.321002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.320974    6695 generic.go:356] "Generic (PLEG): container finished" podID="1276deaf-a385-4f4d-887c-8214e584cf68" containerID="1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf" exitCode=0
Oct 30 22:14:56.321124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.321034    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"
Oct 30 22:14:56.321124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.321054    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" event={"ID":"1276deaf-a385-4f4d-887c-8214e584cf68","Type":"ContainerDied","Data":"1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf"}
Oct 30 22:14:56.321124 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.321109    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7" event={"ID":"1276deaf-a385-4f4d-887c-8214e584cf68","Type":"ContainerDied","Data":"485c788c97d91e5b9a8bff8c4a9ddaba7306b2cbceb386a9d8e280fc924d35be"}
Oct 30 22:14:56.331544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.331494    6695 scope.go:122] "RemoveContainer" containerID="074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61"
Oct 30 22:14:56.331870 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:56.331843    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61\": container with ID starting with 074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61 not found: ID does not exist" containerID="074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61"
Oct 30 22:14:56.331950 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.331881    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61"} err="failed to get container status \"074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61\": rpc error: code = NotFound desc = could not find container \"074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61\": container with ID starting with 074e6d081e2bd1f36f913b277282a47047179c3277a4fe6bcb798548b9615d61 not found: ID does not exist"
Oct 30 22:14:56.331950 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.331922    6695 scope.go:122] "RemoveContainer" containerID="1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf"
Oct 30 22:14:56.341633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.341615    6695 scope.go:122] "RemoveContainer" containerID="1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf"
Oct 30 22:14:56.341914 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:56.341895    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf\": container with ID starting with 1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf not found: ID does not exist" containerID="1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf"
Oct 30 22:14:56.341990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.341925    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf"} err="failed to get container status \"1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf\": rpc error: code = NotFound desc = could not find container \"1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf\": container with ID starting with 1e0bcb5be57628bad95c7d0039b15500e1a2239fcb83194a007d5a2afa703edf not found: ID does not exist"
Oct 30 22:14:56.353967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.353912    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.353967 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.353938    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgx6c\" (UniqueName: \"kubernetes.io/projected/0eb24b4a-0af8-491b-b6bb-7c5fe3353348-kube-api-access-vgx6c\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.371451 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.371416    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"]
Oct 30 22:14:56.376152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.376124    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5494e8-5545f4db7f-bfgb7"]
Oct 30 22:14:56.411143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.411107    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"]
Oct 30 22:14:56.417753 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.417723    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5494e8-67c98745dc-g2qg7"]
Oct 30 22:14:56.458540 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.458513    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:56.555900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.555866    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca\") pod \"b229af07-92f7-424e-89a1-25c699430029\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") "
Oct 30 22:14:56.556084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.555926    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q\" (UniqueName: \"kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q\") pod \"b229af07-92f7-424e-89a1-25c699430029\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") "
Oct 30 22:14:56.556084 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.555967    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca\") pod \"b229af07-92f7-424e-89a1-25c699430029\" (UID: \"b229af07-92f7-424e-89a1-25c699430029\") "
Oct 30 22:14:56.556407 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.556384    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca" pod "b229af07-92f7-424e-89a1-25c699430029" (UID: "b229af07-92f7-424e-89a1-25c699430029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.558164 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.558144    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca" pod "b229af07-92f7-424e-89a1-25c699430029" (UID: "b229af07-92f7-424e-89a1-25c699430029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:56.558253 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.558157    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q" pod "b229af07-92f7-424e-89a1-25c699430029" (UID: "b229af07-92f7-424e-89a1-25c699430029"). InnerVolumeSpecName "kube-api-access-k4p6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:56.658924 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.658822    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b229af07-92f7-424e-89a1-25c699430029-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.658924 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.658863    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4p6q\" (UniqueName: \"kubernetes.io/projected/b229af07-92f7-424e-89a1-25c699430029-kube-api-access-k4p6q\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.658924 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.658886    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b229af07-92f7-424e-89a1-25c699430029-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.924807 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.924745    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 22:14:56.961360 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.961329    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft\" (UniqueName: \"kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft\") pod \"640adee0-832b-4e53-9eb9-22fe76a9f561\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") "
Oct 30 22:14:56.961557 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.961538    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume\") pod \"640adee0-832b-4e53-9eb9-22fe76a9f561\" (UID: \"640adee0-832b-4e53-9eb9-22fe76a9f561\") "
Oct 30 22:14:56.961994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.961965    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume" pod "640adee0-832b-4e53-9eb9-22fe76a9f561" (UID: "640adee0-832b-4e53-9eb9-22fe76a9f561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.963544 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:56.963515    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft" pod "640adee0-832b-4e53-9eb9-22fe76a9f561" (UID: "640adee0-832b-4e53-9eb9-22fe76a9f561"). InnerVolumeSpecName "kube-api-access-h96ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:57.063083 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.063045    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/640adee0-832b-4e53-9eb9-22fe76a9f561-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:57.063083 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.063080    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h96ft\" (UniqueName: \"kubernetes.io/projected/640adee0-832b-4e53-9eb9-22fe76a9f561-kube-api-access-h96ft\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:57.213996 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.213917    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" path="/var/lib/kubelet/pods/0eb24b4a-0af8-491b-b6bb-7c5fe3353348/volumes"
Oct 30 22:14:57.214347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.214333    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" path="/var/lib/kubelet/pods/1276deaf-a385-4f4d-887c-8214e584cf68/volumes"
Oct 30 22:14:57.315966 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.315916    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl" podUID="67835098-f7b5-45f1-bdb6-339d0ba1578e" containerName="payload-pre-processing" containerID="cri-o://8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30" gracePeriod=25
Oct 30 22:14:57.330942 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.330910    6695 generic.go:356] "Generic (PLEG): container finished" podID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerID="601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db" exitCode=0
Oct 30 22:14:57.331333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.330960    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"
Oct 30 22:14:57.331333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.330992    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" event={"ID":"640adee0-832b-4e53-9eb9-22fe76a9f561","Type":"ContainerDied","Data":"601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db"}
Oct 30 22:14:57.331333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.331090    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks" event={"ID":"640adee0-832b-4e53-9eb9-22fe76a9f561","Type":"ContainerDied","Data":"e846d78eca75408991cdf683b8218b3f8fc8dead920ce9018b44eef5726a58a2"}
Oct 30 22:14:57.331333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.331124    6695 scope.go:122] "RemoveContainer" containerID="601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db"
Oct 30 22:14:57.332523 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.332507    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"
Oct 30 22:14:57.332648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.332508    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz" event={"ID":"b229af07-92f7-424e-89a1-25c699430029","Type":"ContainerDied","Data":"91d9945771c5835d999695dfdab3e4261ddf0a4be35dd5bf2429bc500d91441b"}
Oct 30 22:14:57.332648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.332643    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d9945771c5835d999695dfdab3e4261ddf0a4be35dd5bf2429bc500d91441b"
Oct 30 22:14:57.421759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.421728    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"]
Oct 30 22:14:57.429750 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.429641    6695 scope.go:122] "RemoveContainer" containerID="601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db"
Oct 30 22:14:57.430142 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:57.430112    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db\": container with ID starting with 601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db not found: ID does not exist" containerID="601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db"
Oct 30 22:14:57.430216 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.430158    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db"} err="failed to get container status \"601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db\": rpc error: code = NotFound desc = could not find container \"601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db\": container with ID starting with 601759afdeb1084bb5cc9e0da150f49561b1b103ff03a9c6500f09efcf1865db not found: ID does not exist"
Oct 30 22:14:57.433732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.433706    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e06b90-579cd58d7f-czsks"]
Oct 30 22:14:57.505050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.505015    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:57.505194 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.505046    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:57.557475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.557452    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 22:14:57.669639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.669582    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85\") pod \"67835098-f7b5-45f1-bdb6-339d0ba1578e\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") "
Oct 30 22:14:57.669639 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.669635    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume\") pod \"67835098-f7b5-45f1-bdb6-339d0ba1578e\" (UID: \"67835098-f7b5-45f1-bdb6-339d0ba1578e\") "
Oct 30 22:14:57.670022 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.669993    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume" pod "67835098-f7b5-45f1-bdb6-339d0ba1578e" (UID: "67835098-f7b5-45f1-bdb6-339d0ba1578e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:57.671794 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.671761    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85" pod "67835098-f7b5-45f1-bdb6-339d0ba1578e" (UID: "67835098-f7b5-45f1-bdb6-339d0ba1578e"). InnerVolumeSpecName "kube-api-access-2pd85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:57.771442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.771413    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pd85\" (UniqueName: \"kubernetes.io/projected/67835098-f7b5-45f1-bdb6-339d0ba1578e-kube-api-access-2pd85\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:57.771442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:57.771444    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67835098-f7b5-45f1-bdb6-339d0ba1578e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:14:58.342309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.342272    6695 generic.go:356] "Generic (PLEG): container finished" podID="67835098-f7b5-45f1-bdb6-339d0ba1578e" containerID="8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30" exitCode=0
Oct 30 22:14:58.342752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.342319    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"
Oct 30 22:14:58.342752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.342358    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl" event={"ID":"67835098-f7b5-45f1-bdb6-339d0ba1578e","Type":"ContainerDied","Data":"8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30"}
Oct 30 22:14:58.342752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.342410    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl" event={"ID":"67835098-f7b5-45f1-bdb6-339d0ba1578e","Type":"ContainerDied","Data":"8219b84c52abd1ecbc5195cab0ab34b7a685de022d767a01ba555981faa40e4d"}
Oct 30 22:14:58.342752 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.342432    6695 scope.go:122] "RemoveContainer" containerID="8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30"
Oct 30 22:14:58.352307 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.352289    6695 scope.go:122] "RemoveContainer" containerID="8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30"
Oct 30 22:14:58.352563 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:14:58.352542    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30\": container with ID starting with 8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30 not found: ID does not exist" containerID="8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30"
Oct 30 22:14:58.352693 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.352570    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30"} err="failed to get container status \"8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30\": rpc error: code = NotFound desc = could not find container \"8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30\": container with ID starting with 8a79be3eda2d14d46c7e10c917ce67c3b6a7ba6fb0260a24596a03e31310eb30 not found: ID does not exist"
Oct 30 22:14:58.397676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.397646    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"]
Oct 30 22:14:58.408208 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.408185    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-ae6492-5476fdd555-qnnzl"]
Oct 30 22:14:58.410301 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.410276    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"]
Oct 30 22:14:58.412094 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:58.412075    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ae6492-lkp8z"]
Oct 30 22:14:59.213631 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:59.213533    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" path="/var/lib/kubelet/pods/640adee0-832b-4e53-9eb9-22fe76a9f561/volumes"
Oct 30 22:14:59.213981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:59.213965    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67835098-f7b5-45f1-bdb6-339d0ba1578e" path="/var/lib/kubelet/pods/67835098-f7b5-45f1-bdb6-339d0ba1578e/volumes"
Oct 30 22:14:59.214341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:14:59.214329    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c40ced3-974b-4945-ad12-87dd51f84222" path="/var/lib/kubelet/pods/7c40ced3-974b-4945-ad12-87dd51f84222/volumes"
Oct 30 22:15:00.144746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.144713    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"]
Oct 30 22:15:00.145175 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145161    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerName="payload-pre-processing"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145176    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerName="payload-pre-processing"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145191    6695 cpu_manager.go:414] "containerMap: removing container" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145197    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c4c2b4-4a4b-41d6-aa81-66f0da709615" containerName="istio-proxy"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145211    6695 cpu_manager.go:414] "containerMap: removing container" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerName="payload-pre-processing"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145216    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerName="payload-pre-processing"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145224    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b229af07-92f7-424e-89a1-25c699430029" containerName="revoke-keys"
Oct 30 22:15:00.145237 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145231    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b229af07-92f7-424e-89a1-25c699430029" containerName="revoke-keys"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145247    6695 cpu_manager.go:414] "containerMap: removing container" podUID="67835098-f7b5-45f1-bdb6-339d0ba1578e" containerName="payload-pre-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145252    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67835098-f7b5-45f1-bdb6-339d0ba1578e" containerName="payload-pre-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145277    6695 cpu_manager.go:414] "containerMap: removing container" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" containerName="payload-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145284    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" containerName="payload-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145355    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="67835098-f7b5-45f1-bdb6-339d0ba1578e" containerName="payload-pre-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145365    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="1276deaf-a385-4f4d-887c-8214e584cf68" containerName="payload-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145374    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb24b4a-0af8-491b-b6bb-7c5fe3353348" containerName="payload-pre-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145382    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="640adee0-832b-4e53-9eb9-22fe76a9f561" containerName="payload-pre-processing"
Oct 30 22:15:00.145533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.145388    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b229af07-92f7-424e-89a1-25c699430029" containerName="revoke-keys"
Oct 30 22:15:00.150116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.150096    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.158128 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.158100    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"]
Oct 30 22:15:00.162419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.162396    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"]
Oct 30 22:15:00.162530 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.162499    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.166690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.166662    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.170797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.170775    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:00.170937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.170819    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:00.173913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.173888    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"]
Oct 30 22:15:00.182720 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.182698    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:15:00.193574 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.193536    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"]
Oct 30 22:15:00.196192 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.196165    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"]
Oct 30 22:15:00.203116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.203084    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5w9j\" (UniqueName: \"kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.203265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.203149    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvsz\" (UniqueName: \"kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.203265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.203240    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-t65nt\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.203365 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.203297    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.203493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.203470    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.203570 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.203554    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b\") pod \"maas-api-key-cleanup-29889975-t65nt\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.227323 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.227284    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"]
Oct 30 22:15:00.227638 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:00.227596    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-w5w9j maas-api-service-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7" podUID="a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3"
Oct 30 22:15:00.304358 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.304317    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.304573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.304393    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b\") pod \"maas-api-key-cleanup-29889975-t65nt\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.304670 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.304643    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5w9j\" (UniqueName: \"kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.304812 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.304790    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvsz\" (UniqueName: \"kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.304897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.304839    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-t65nt\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.304963 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.304929    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.305186 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.305163    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.305454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.305436    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-t65nt\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.305521 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.305505    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.328732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.328708    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5w9j\" (UniqueName: \"kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.328732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.328724    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b\") pod \"maas-api-key-cleanup-29889975-t65nt\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.329050 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.329032    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvsz\" (UniqueName: \"kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.355413 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.355388    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.364325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.364302    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:00.406079 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.405982    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j\" (UniqueName: \"kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j\") pod \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") "
Oct 30 22:15:00.406232 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.406104    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca\") pod \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\" (UID: \"a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3\") "
Oct 30 22:15:00.406779 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.406744    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca" pod "a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3" (UID: "a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:00.408337 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.408307    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j" pod "a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3" (UID: "a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3"). InnerVolumeSpecName "kube-api-access-w5w9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:00.466686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.466655    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:00.485364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.485326    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:00.508265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.508229    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5w9j\" (UniqueName: \"kubernetes.io/projected/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-kube-api-access-w5w9j\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:00.508389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.508270    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:00.788907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.788862    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"]
Oct 30 22:15:00.792642 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:15:00.792593    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3f3fcc_9681_490b_9372_fd4e7f081385.slice/crio-570730069f35f0c29d71dba63f1f9baa7aa5f5eff59a48427d5fccd17322080a WatchSource:0}: Error finding container 570730069f35f0c29d71dba63f1f9baa7aa5f5eff59a48427d5fccd17322080a: Status 404 returned error can't find the container with id 570730069f35f0c29d71dba63f1f9baa7aa5f5eff59a48427d5fccd17322080a
Oct 30 22:15:00.852596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.852566    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"]
Oct 30 22:15:00.856031 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:15:00.856003    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a30ad7_2868_41bc_becb_25b199488b2c.slice/crio-ed46e77af02eefdf1c830878ecd6b0c5d733d15410585a8b1310db68859fcddb WatchSource:0}: Error finding container ed46e77af02eefdf1c830878ecd6b0c5d733d15410585a8b1310db68859fcddb: Status 404 returned error can't find the container with id ed46e77af02eefdf1c830878ecd6b0c5d733d15410585a8b1310db68859fcddb
Oct 30 22:15:00.922819 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.922782    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"]
Oct 30 22:15:00.937062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:00.937030    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b31518-tdbhx"]
Oct 30 22:15:01.074164 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:01.074077    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:15:01.223100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.223075    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3443d6a4-2241-48fe-ad9d-e3910403d27b" path="/var/lib/kubelet/pods/3443d6a4-2241-48fe-ad9d-e3910403d27b/volumes"
Oct 30 22:15:01.367045 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.367002    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" event={"ID":"29a30ad7-2868-41bc-becb-25b199488b2c","Type":"ContainerStarted","Data":"3aa900515c87aa5c430e9699117d288b67c628aaa19688e859ad1644914a3012"}
Oct 30 22:15:01.367246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.367078    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" event={"ID":"29a30ad7-2868-41bc-becb-25b199488b2c","Type":"ContainerStarted","Data":"ed46e77af02eefdf1c830878ecd6b0c5d733d15410585a8b1310db68859fcddb"}
Oct 30 22:15:01.368841 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.368810    6695 generic.go:356] "Generic (PLEG): container finished" podID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerID="e2e68297cbc3556a5e294773124501153ff096c21bf67f90b5ab0f3400693b03" exitCode=0
Oct 30 22:15:01.368980 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.368911    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" event={"ID":"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643","Type":"ContainerDied","Data":"e2e68297cbc3556a5e294773124501153ff096c21bf67f90b5ab0f3400693b03"}
Oct 30 22:15:01.371459 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.371429    6695 generic.go:356] "Generic (PLEG): container finished" podID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerID="0de39b3c68e20a4c098a17370f36b86bac13833719ca71ed98e2275409d5dd41" exitCode=0
Oct 30 22:15:01.371574 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.371509    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" event={"ID":"960d0dfc-ea23-4db4-9a2c-12b459af443e","Type":"ContainerDied","Data":"0de39b3c68e20a4c098a17370f36b86bac13833719ca71ed98e2275409d5dd41"}
Oct 30 22:15:01.373494 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.373472    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"
Oct 30 22:15:01.374388 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.374366    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" event={"ID":"5c3f3fcc-9681-490b-9372-fd4e7f081385","Type":"ContainerStarted","Data":"0ad0db93f7a33b1bdf1b923ddcbee82368528408282c3b2eb4e277e1cad8970a"}
Oct 30 22:15:01.374515 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.374410    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" event={"ID":"5c3f3fcc-9681-490b-9372-fd4e7f081385","Type":"ContainerStarted","Data":"570730069f35f0c29d71dba63f1f9baa7aa5f5eff59a48427d5fccd17322080a"}
Oct 30 22:15:01.425813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.425760    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" podStartSLOduration=1.425743251 podStartE2EDuration="1.425743251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.422470324 +0000 UTC m=+3884.344928621" watchObservedRunningTime="2026-10-30 22:15:01.425743251 +0000 UTC m=+3884.348201526"
Oct 30 22:15:01.447548 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.447517    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 22:15:01.481958 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.481932    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 22:15:01.490660 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.490624    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"]
Oct 30 22:15:01.498238 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.498204    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889975-qfpk7"]
Oct 30 22:15:01.522034 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.522001    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx\" (UniqueName: \"kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx\") pod \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") "
Oct 30 22:15:01.522202 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.522088    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume\") pod \"960d0dfc-ea23-4db4-9a2c-12b459af443e\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") "
Oct 30 22:15:01.522202 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.522176    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume\") pod \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\" (UID: \"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643\") "
Oct 30 22:15:01.522202 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.522196    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s\" (UniqueName: \"kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s\") pod \"960d0dfc-ea23-4db4-9a2c-12b459af443e\" (UID: \"960d0dfc-ea23-4db4-9a2c-12b459af443e\") "
Oct 30 22:15:01.522533 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.522504    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume" pod "960d0dfc-ea23-4db4-9a2c-12b459af443e" (UID: "960d0dfc-ea23-4db4-9a2c-12b459af443e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:01.522622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.522551    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume" pod "2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" (UID: "2c4bc7cf-96c4-4da2-ae1e-e4e99e554643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:01.524055 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.524032    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx" pod "2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" (UID: "2c4bc7cf-96c4-4da2-ae1e-e4e99e554643"). InnerVolumeSpecName "kube-api-access-2xmwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:01.524163 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.524149    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s" pod "960d0dfc-ea23-4db4-9a2c-12b459af443e" (UID: "960d0dfc-ea23-4db4-9a2c-12b459af443e"). InnerVolumeSpecName "kube-api-access-vhg8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:01.537845 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.537798    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" podStartSLOduration=1.53778266 podStartE2EDuration="1.53778266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:01.536452838 +0000 UTC m=+3884.458911135" watchObservedRunningTime="2026-10-30 22:15:01.53778266 +0000 UTC m=+3884.460240960"
Oct 30 22:15:01.623890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.623797    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:01.623890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.623830    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhg8s\" (UniqueName: \"kubernetes.io/projected/960d0dfc-ea23-4db4-9a2c-12b459af443e-kube-api-access-vhg8s\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:01.623890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.623843    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmwx\" (UniqueName: \"kubernetes.io/projected/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643-kube-api-access-2xmwx\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:01.623890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:01.623856    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d0dfc-ea23-4db4-9a2c-12b459af443e-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:02.381280 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.381244    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"
Oct 30 22:15:02.381746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.381251    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242" event={"ID":"960d0dfc-ea23-4db4-9a2c-12b459af443e","Type":"ContainerDied","Data":"acea1d8d0e13c2d1fc72a1d28b1d34b9d6ef4f1e9014afceade09bbc90c69859"}
Oct 30 22:15:02.381746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.381405    6695 scope.go:122] "RemoveContainer" containerID="0de39b3c68e20a4c098a17370f36b86bac13833719ca71ed98e2275409d5dd41"
Oct 30 22:15:02.383132 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.383096    6695 generic.go:356] "Generic (PLEG): container finished" podID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerID="0ad0db93f7a33b1bdf1b923ddcbee82368528408282c3b2eb4e277e1cad8970a" exitCode=60
Oct 30 22:15:02.383271 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.383183    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" event={"ID":"5c3f3fcc-9681-490b-9372-fd4e7f081385","Type":"ContainerDied","Data":"0ad0db93f7a33b1bdf1b923ddcbee82368528408282c3b2eb4e277e1cad8970a"}
Oct 30 22:15:02.383457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.383438    6695 scope.go:122] "RemoveContainer" containerID="0ad0db93f7a33b1bdf1b923ddcbee82368528408282c3b2eb4e277e1cad8970a"
Oct 30 22:15:02.385105 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.385082    6695 generic.go:356] "Generic (PLEG): container finished" podID="29a30ad7-2868-41bc-becb-25b199488b2c" containerID="3aa900515c87aa5c430e9699117d288b67c628aaa19688e859ad1644914a3012" exitCode=60
Oct 30 22:15:02.385183 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.385167    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" event={"ID":"29a30ad7-2868-41bc-becb-25b199488b2c","Type":"ContainerDied","Data":"3aa900515c87aa5c430e9699117d288b67c628aaa19688e859ad1644914a3012"}
Oct 30 22:15:02.385400 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.385375    6695 scope.go:122] "RemoveContainer" containerID="3aa900515c87aa5c430e9699117d288b67c628aaa19688e859ad1644914a3012"
Oct 30 22:15:02.386889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.386715    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp" event={"ID":"2c4bc7cf-96c4-4da2-ae1e-e4e99e554643","Type":"ContainerDied","Data":"03e35a6ced9a6519a1374f1059c52baaffc939956c23e8d3efe8e745cdb733d7"}
Oct 30 22:15:02.386889 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.386722    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"
Oct 30 22:15:02.400152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.400131    6695 scope.go:122] "RemoveContainer" containerID="e2e68297cbc3556a5e294773124501153ff096c21bf67f90b5ab0f3400693b03"
Oct 30 22:15:02.490181 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.490140    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"]
Oct 30 22:15:02.497312 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.497286    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b31518-67c4fb9fcf-nz242"]
Oct 30 22:15:02.572510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.571458    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"]
Oct 30 22:15:02.573676 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:02.573645    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b31518-ccb5bcdf7-dzthp"]
Oct 30 22:15:03.213143 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.213108    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" path="/var/lib/kubelet/pods/2c4bc7cf-96c4-4da2-ae1e-e4e99e554643/volumes"
Oct 30 22:15:03.213585 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.213564    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" path="/var/lib/kubelet/pods/960d0dfc-ea23-4db4-9a2c-12b459af443e/volumes"
Oct 30 22:15:03.214024 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.214006    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3" path="/var/lib/kubelet/pods/a2b83d5a-6c2f-44ba-ae72-77d315e9c9e3/volumes"
Oct 30 22:15:03.395713 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.395676    6695 generic.go:356] "Generic (PLEG): container finished" podID="29a30ad7-2868-41bc-becb-25b199488b2c" containerID="c5221b7e4566480684a214c1e8e8be7e067583d99ec45dea1a41439688ba5a40" exitCode=60
Oct 30 22:15:03.396218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.395720    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" event={"ID":"29a30ad7-2868-41bc-becb-25b199488b2c","Type":"ContainerDied","Data":"c5221b7e4566480684a214c1e8e8be7e067583d99ec45dea1a41439688ba5a40"}
Oct 30 22:15:03.396218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.395779    6695 scope.go:122] "RemoveContainer" containerID="3aa900515c87aa5c430e9699117d288b67c628aaa19688e859ad1644914a3012"
Oct 30 22:15:03.396218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.396036    6695 scope.go:122] "RemoveContainer" containerID="c5221b7e4566480684a214c1e8e8be7e067583d99ec45dea1a41439688ba5a40"
Oct 30 22:15:03.396408 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:03.396337    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56_odh-ai-gateway-infra(29a30ad7-2868-41bc-becb-25b199488b2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" podUID="29a30ad7-2868-41bc-becb-25b199488b2c"
Oct 30 22:15:03.400091 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.400065    6695 generic.go:356] "Generic (PLEG): container finished" podID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerID="20928b44e0963e385393d9ad6c48a98442f18c1d6917d407821a1e55272f152e" exitCode=60
Oct 30 22:15:03.400221 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.400153    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" event={"ID":"5c3f3fcc-9681-490b-9372-fd4e7f081385","Type":"ContainerDied","Data":"20928b44e0963e385393d9ad6c48a98442f18c1d6917d407821a1e55272f152e"}
Oct 30 22:15:03.400458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.400442    6695 scope.go:122] "RemoveContainer" containerID="20928b44e0963e385393d9ad6c48a98442f18c1d6917d407821a1e55272f152e"
Oct 30 22:15:03.400720 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:03.400700    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-t65nt_odh-ai-gateway-infra(5c3f3fcc-9681-490b-9372-fd4e7f081385)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385"
Oct 30 22:15:03.408567 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.408549    6695 scope.go:122] "RemoveContainer" containerID="0ad0db93f7a33b1bdf1b923ddcbee82368528408282c3b2eb4e277e1cad8970a"
Oct 30 22:15:03.978744 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:03.978706    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"]
Oct 30 22:15:04.016721 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:04.016685    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"]
Oct 30 22:15:04.409813 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:04.409779    6695 scope.go:122] "RemoveContainer" containerID="c5221b7e4566480684a214c1e8e8be7e067583d99ec45dea1a41439688ba5a40"
Oct 30 22:15:04.410308 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:04.410053    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56_odh-ai-gateway-infra(29a30ad7-2868-41bc-becb-25b199488b2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" podUID="29a30ad7-2868-41bc-becb-25b199488b2c"
Oct 30 22:15:04.411307 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:04.411289    6695 scope.go:122] "RemoveContainer" containerID="20928b44e0963e385393d9ad6c48a98442f18c1d6917d407821a1e55272f152e"
Oct 30 22:15:04.411506 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:04.411487    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-t65nt_odh-ai-gateway-infra(5c3f3fcc-9681-490b-9372-fd4e7f081385)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385"
Oct 30 22:15:06.062215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.062179    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"]
Oct 30 22:15:06.073598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.073564    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3347982f-q54dv"]
Oct 30 22:15:06.076042 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.076014    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"]
Oct 30 22:15:06.084873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.084837    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa173de0-hldxw"]
Oct 30 22:15:06.804621 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.804571    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"]
Oct 30 22:15:06.805068 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.805053    6695 cpu_manager.go:414] "containerMap: removing container" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerName="payload-processing"
Oct 30 22:15:06.805127 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.805068    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerName="payload-processing"
Oct 30 22:15:06.805127 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.805090    6695 cpu_manager.go:414] "containerMap: removing container" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerName="payload-pre-processing"
Oct 30 22:15:06.805127 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.805095    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerName="payload-pre-processing"
Oct 30 22:15:06.805239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.805156    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="960d0dfc-ea23-4db4-9a2c-12b459af443e" containerName="payload-pre-processing"
Oct 30 22:15:06.805239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.805170    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4bc7cf-96c4-4da2-ae1e-e4e99e554643" containerName="payload-processing"
Oct 30 22:15:06.809517 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.809500    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.836133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.836101    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"]
Oct 30 22:15:06.881665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.881630    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrc9\" (UniqueName: \"kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.881811 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.881776    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.881909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.881894    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.901495 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.901473    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:06.982680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.982648    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.982878 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.982740    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.982878 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.982772    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrc9\" (UniqueName: \"kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.983434 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.983413    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:06.985035 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:06.985016    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:07.002120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:07.002095    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrc9\" (UniqueName: \"kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9\") pod \"maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:07.123190 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:07.123106    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:07.215622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:07.215579    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f188c9f3-6f8b-41b0-950c-ddec78c802ea" path="/var/lib/kubelet/pods/f188c9f3-6f8b-41b0-950c-ddec78c802ea/volumes"
Oct 30 22:15:07.216052 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:07.216037    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84c686f-3e63-4583-b56e-917e52c1db50" path="/var/lib/kubelet/pods/f84c686f-3e63-4583-b56e-917e52c1db50/volumes"
Oct 30 22:15:07.428661 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:07.428544    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"]
Oct 30 22:15:07.433339 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:15:07.433306    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ba3710_8177_43a3_99f6_b612a352c2df.slice/crio-17f914693d86cb44ff8953544c64bf734e78f72d53ed12496c463ffdc5629065 WatchSource:0}: Error finding container 17f914693d86cb44ff8953544c64bf734e78f72d53ed12496c463ffdc5629065: Status 404 returned error can't find the container with id 17f914693d86cb44ff8953544c64bf734e78f72d53ed12496c463ffdc5629065
Oct 30 22:15:08.446166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:08.446125    6695 generic.go:356] "Generic (PLEG): container finished" podID="d7ba3710-8177-43a3-99f6-b612a352c2df" containerID="1631371b36aa42d09412f1d50356dd89233ab52da0f6379d9bc6018386c154aa" exitCode=0
Oct 30 22:15:08.446591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:08.446210    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc" event={"ID":"d7ba3710-8177-43a3-99f6-b612a352c2df","Type":"ContainerDied","Data":"1631371b36aa42d09412f1d50356dd89233ab52da0f6379d9bc6018386c154aa"}
Oct 30 22:15:08.446591 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:08.446280    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc" event={"ID":"d7ba3710-8177-43a3-99f6-b612a352c2df","Type":"ContainerStarted","Data":"17f914693d86cb44ff8953544c64bf734e78f72d53ed12496c463ffdc5629065"}
Oct 30 22:15:08.590107 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:08.590023    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"]
Oct 30 22:15:08.593199 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:08.593156    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-64d142-sxhrz"]
Oct 30 22:15:08.979705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:08.979570    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerName="payload-processing" containerID="cri-o://84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8" gracePeriod=25
Oct 30 22:15:09.017275 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.017231    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerName="payload-pre-processing" containerID="cri-o://0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2" gracePeriod=25
Oct 30 22:15:09.212537 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.212498    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b229af07-92f7-424e-89a1-25c699430029" path="/var/lib/kubelet/pods/b229af07-92f7-424e-89a1-25c699430029/volumes"
Oct 30 22:15:09.321133 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.321090    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:09.321377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.321347    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:09.615816 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.615781    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:09.713418 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.713374    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca\") pod \"d7ba3710-8177-43a3-99f6-b612a352c2df\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") "
Oct 30 22:15:09.713627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.713477    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9\" (UniqueName: \"kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9\") pod \"d7ba3710-8177-43a3-99f6-b612a352c2df\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") "
Oct 30 22:15:09.713627 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.713562    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca\") pod \"d7ba3710-8177-43a3-99f6-b612a352c2df\" (UID: \"d7ba3710-8177-43a3-99f6-b612a352c2df\") "
Oct 30 22:15:09.713882 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.713849    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca" pod "d7ba3710-8177-43a3-99f6-b612a352c2df" (UID: "d7ba3710-8177-43a3-99f6-b612a352c2df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:09.713973 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.713945    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:09.715805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.715780    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9" pod "d7ba3710-8177-43a3-99f6-b612a352c2df" (UID: "d7ba3710-8177-43a3-99f6-b612a352c2df"). InnerVolumeSpecName "kube-api-access-pbrc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:09.716152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.716128    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca" pod "d7ba3710-8177-43a3-99f6-b612a352c2df" (UID: "d7ba3710-8177-43a3-99f6-b612a352c2df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:09.821303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.821254    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrc9\" (UniqueName: \"kubernetes.io/projected/d7ba3710-8177-43a3-99f6-b612a352c2df-kube-api-access-pbrc9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:09.821303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:09.821297    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ba3710-8177-43a3-99f6-b612a352c2df-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:10.460325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:10.460283    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc" event={"ID":"d7ba3710-8177-43a3-99f6-b612a352c2df","Type":"ContainerDied","Data":"17f914693d86cb44ff8953544c64bf734e78f72d53ed12496c463ffdc5629065"}
Oct 30 22:15:10.460510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:10.460352    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17f914693d86cb44ff8953544c64bf734e78f72d53ed12496c463ffdc5629065"
Oct 30 22:15:10.460510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:10.460299    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"
Oct 30 22:15:11.511799 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:11.511764    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"]
Oct 30 22:15:11.525686 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:11.525658    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"]
Oct 30 22:15:15.206664 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:15.206632    6695 scope.go:122] "RemoveContainer" containerID="c5221b7e4566480684a214c1e8e8be7e067583d99ec45dea1a41439688ba5a40"
Oct 30 22:15:16.499994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:16.499955    6695 generic.go:356] "Generic (PLEG): container finished" podID="29a30ad7-2868-41bc-becb-25b199488b2c" containerID="480c4bf5689f208af68ff771779ae37c94f66875078c6fea31ef2805ba8a54e9" exitCode=60
Oct 30 22:15:16.500444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:16.500029    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" event={"ID":"29a30ad7-2868-41bc-becb-25b199488b2c","Type":"ContainerDied","Data":"480c4bf5689f208af68ff771779ae37c94f66875078c6fea31ef2805ba8a54e9"}
Oct 30 22:15:16.500444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:16.500091    6695 scope.go:122] "RemoveContainer" containerID="c5221b7e4566480684a214c1e8e8be7e067583d99ec45dea1a41439688ba5a40"
Oct 30 22:15:16.500444 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:16.500438    6695 scope.go:122] "RemoveContainer" containerID="480c4bf5689f208af68ff771779ae37c94f66875078c6fea31ef2805ba8a54e9"
Oct 30 22:15:16.500737 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:16.500717    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56_odh-ai-gateway-infra(29a30ad7-2868-41bc-becb-25b199488b2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" podUID="29a30ad7-2868-41bc-becb-25b199488b2c"
Oct 30 22:15:17.215267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.215234    6695 scope.go:122] "RemoveContainer" containerID="20928b44e0963e385393d9ad6c48a98442f18c1d6917d407821a1e55272f152e"
Oct 30 22:15:17.371512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.371461    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"]
Oct 30 22:15:17.372789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.372760    6695 cpu_manager.go:414] "containerMap: removing container" podUID="d7ba3710-8177-43a3-99f6-b612a352c2df" containerName="revoke-keys"
Oct 30 22:15:17.372985 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.372970    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ba3710-8177-43a3-99f6-b612a352c2df" containerName="revoke-keys"
Oct 30 22:15:17.373340 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.373316    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ba3710-8177-43a3-99f6-b612a352c2df" containerName="revoke-keys"
Oct 30 22:15:17.378263 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.378241    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.394640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.394027    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"]
Oct 30 22:15:17.402509 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.402479    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:17.513948 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.513855    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.514411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.514030    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.514411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.514078    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr26f\" (UniqueName: \"kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.561353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.561319    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"]
Oct 30 22:15:17.615152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.615123    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr26f\" (UniqueName: \"kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.615516 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.615492    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.616120 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.616094    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.616291 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.616270    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.629871 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.629828    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:17.642241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.640057    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.642241 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.640233    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr26f\" (UniqueName: \"kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f\") pod \"maas-api-revoke-sub-4922e3af-v7rgd\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.693006 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.692977    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:17.710564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.710537    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:17.818016 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.817967    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca\") pod \"29a30ad7-2868-41bc-becb-25b199488b2c\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") "
Oct 30 22:15:17.818227 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.818039    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz\" (UniqueName: \"kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz\") pod \"29a30ad7-2868-41bc-becb-25b199488b2c\" (UID: \"29a30ad7-2868-41bc-becb-25b199488b2c\") "
Oct 30 22:15:17.818442 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.818413    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca" pod "29a30ad7-2868-41bc-becb-25b199488b2c" (UID: "29a30ad7-2868-41bc-becb-25b199488b2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:17.820462 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.820433    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz" pod "29a30ad7-2868-41bc-becb-25b199488b2c" (UID: "29a30ad7-2868-41bc-becb-25b199488b2c"). InnerVolumeSpecName "kube-api-access-zdvsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:17.919648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.919589    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29a30ad7-2868-41bc-becb-25b199488b2c-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:17.919648 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:17.919644    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdvsz\" (UniqueName: \"kubernetes.io/projected/29a30ad7-2868-41bc-becb-25b199488b2c-kube-api-access-zdvsz\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:18.009775 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.009728    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"]
Oct 30 22:15:18.013954 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:15:18.013920    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34f18ae_421b_4a0b_b069_c174f51a1e97.slice/crio-461716f0a50510ebdae2a83a47aa5f2912a499e8ac8f13f1e50b29fcd44d6cd0 WatchSource:0}: Error finding container 461716f0a50510ebdae2a83a47aa5f2912a499e8ac8f13f1e50b29fcd44d6cd0: Status 404 returned error can't find the container with id 461716f0a50510ebdae2a83a47aa5f2912a499e8ac8f13f1e50b29fcd44d6cd0
Oct 30 22:15:18.517539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.517453    6695 generic.go:356] "Generic (PLEG): container finished" podID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerID="460f7c5f5a1ed1b3bd067ca4cd38691fabf94426824f2d052f327fe6a5402213" exitCode=60
Oct 30 22:15:18.517539 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.517526    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" event={"ID":"5c3f3fcc-9681-490b-9372-fd4e7f081385","Type":"ContainerDied","Data":"460f7c5f5a1ed1b3bd067ca4cd38691fabf94426824f2d052f327fe6a5402213"}
Oct 30 22:15:18.518062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.517586    6695 scope.go:122] "RemoveContainer" containerID="20928b44e0963e385393d9ad6c48a98442f18c1d6917d407821a1e55272f152e"
Oct 30 22:15:18.518062 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.517971    6695 scope.go:122] "RemoveContainer" containerID="460f7c5f5a1ed1b3bd067ca4cd38691fabf94426824f2d052f327fe6a5402213"
Oct 30 22:15:18.518293 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:18.518270    6695 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-t65nt_odh-ai-gateway-infra(5c3f3fcc-9681-490b-9372-fd4e7f081385)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385"
Oct 30 22:15:18.519457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.519433    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd" event={"ID":"b34f18ae-421b-4a0b-b069-c174f51a1e97","Type":"ContainerStarted","Data":"df9cbc0e648a2f5633a280869d847c2117913e6abe1be2ff404c445b59dbf8f6"}
Oct 30 22:15:18.519562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.519478    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd" event={"ID":"b34f18ae-421b-4a0b-b069-c174f51a1e97","Type":"ContainerStarted","Data":"461716f0a50510ebdae2a83a47aa5f2912a499e8ac8f13f1e50b29fcd44d6cd0"}
Oct 30 22:15:18.520788 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.520772    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"
Oct 30 22:15:18.520913 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.520815    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56" event={"ID":"29a30ad7-2868-41bc-becb-25b199488b2c","Type":"ContainerDied","Data":"ed46e77af02eefdf1c830878ecd6b0c5d733d15410585a8b1310db68859fcddb"}
Oct 30 22:15:18.530395 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.530377    6695 scope.go:122] "RemoveContainer" containerID="480c4bf5689f208af68ff771779ae37c94f66875078c6fea31ef2805ba8a54e9"
Oct 30 22:15:18.592956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.592924    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"]
Oct 30 22:15:18.610025 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.609997    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889975-7dl56"]
Oct 30 22:15:18.655184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:18.655127    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd" podStartSLOduration=1.655114287 podStartE2EDuration="1.655114287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:18.65300992 +0000 UTC m=+3901.575468218" watchObservedRunningTime="2026-10-30 22:15:18.655114287 +0000 UTC m=+3901.577572586"
Oct 30 22:15:19.212478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.212396    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" path="/var/lib/kubelet/pods/29a30ad7-2868-41bc-becb-25b199488b2c/volumes"
Oct 30 22:15:19.317178 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.317133    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:19.317674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.317641    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:19.531735 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.531699    6695 generic.go:356] "Generic (PLEG): container finished" podID="b34f18ae-421b-4a0b-b069-c174f51a1e97" containerID="df9cbc0e648a2f5633a280869d847c2117913e6abe1be2ff404c445b59dbf8f6" exitCode=0
Oct 30 22:15:19.532154 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.531779    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd" event={"ID":"b34f18ae-421b-4a0b-b069-c174f51a1e97","Type":"ContainerDied","Data":"df9cbc0e648a2f5633a280869d847c2117913e6abe1be2ff404c445b59dbf8f6"}
Oct 30 22:15:19.583468 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.583435    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"]
Oct 30 22:15:19.739808 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.739782    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:19.844709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.844624    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b\") pod \"5c3f3fcc-9681-490b-9372-fd4e7f081385\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") "
Oct 30 22:15:19.844709 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.844680    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca\") pod \"5c3f3fcc-9681-490b-9372-fd4e7f081385\" (UID: \"5c3f3fcc-9681-490b-9372-fd4e7f081385\") "
Oct 30 22:15:19.845068 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.845047    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca" pod "5c3f3fcc-9681-490b-9372-fd4e7f081385" (UID: "5c3f3fcc-9681-490b-9372-fd4e7f081385"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.846771 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.846747    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b" pod "5c3f3fcc-9681-490b-9372-fd4e7f081385" (UID: "5c3f3fcc-9681-490b-9372-fd4e7f081385"). InnerVolumeSpecName "kube-api-access-wfs9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.945817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.945781    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/5c3f3fcc-9681-490b-9372-fd4e7f081385-kube-api-access-wfs9b\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:19.945817 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:19.945812    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3f3fcc-9681-490b-9372-fd4e7f081385-maas-api-service-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:20.540764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.540729    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"
Oct 30 22:15:20.540764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.540745    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt" event={"ID":"5c3f3fcc-9681-490b-9372-fd4e7f081385","Type":"ContainerDied","Data":"570730069f35f0c29d71dba63f1f9baa7aa5f5eff59a48427d5fccd17322080a"}
Oct 30 22:15:20.541333 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.540815    6695 scope.go:122] "RemoveContainer" containerID="460f7c5f5a1ed1b3bd067ca4cd38691fabf94426824f2d052f327fe6a5402213"
Oct 30 22:15:20.595576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.595541    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"]
Oct 30 22:15:20.619244 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.619209    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-t65nt"]
Oct 30 22:15:20.691269 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.691242    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:20.817117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.817033    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 22:15:20.848403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.848377    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x52r2_054fd709-2d73-4ce0-ab94-0df9fbd1921a/ovn-acl-logging/0.log"
Oct 30 22:15:20.855080 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.855053    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f\" (UniqueName: \"kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f\") pod \"b34f18ae-421b-4a0b-b069-c174f51a1e97\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") "
Oct 30 22:15:20.855266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.855109    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca\") pod \"b34f18ae-421b-4a0b-b069-c174f51a1e97\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") "
Oct 30 22:15:20.855266 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.855163    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca\") pod \"b34f18ae-421b-4a0b-b069-c174f51a1e97\" (UID: \"b34f18ae-421b-4a0b-b069-c174f51a1e97\") "
Oct 30 22:15:20.855681 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.855650    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca" pod "b34f18ae-421b-4a0b-b069-c174f51a1e97" (UID: "b34f18ae-421b-4a0b-b069-c174f51a1e97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.858792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.858767    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca" pod "b34f18ae-421b-4a0b-b069-c174f51a1e97" (UID: "b34f18ae-421b-4a0b-b069-c174f51a1e97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:20.858792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.858780    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f" pod "b34f18ae-421b-4a0b-b069-c174f51a1e97" (UID: "b34f18ae-421b-4a0b-b069-c174f51a1e97"). InnerVolumeSpecName "kube-api-access-pr26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.957338 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.957286    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr26f\" (UniqueName: \"kubernetes.io/projected/b34f18ae-421b-4a0b-b069-c174f51a1e97-kube-api-access-pr26f\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:20.957563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.957356    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:20.957563 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:20.957378    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34f18ae-421b-4a0b-b069-c174f51a1e97-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.213230 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:21.213153    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" path="/var/lib/kubelet/pods/5c3f3fcc-9681-490b-9372-fd4e7f081385/volumes"
Oct 30 22:15:21.551929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:21.551898    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd"
Oct 30 22:15:21.552351 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:21.551901    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4922e3af-v7rgd" event={"ID":"b34f18ae-421b-4a0b-b069-c174f51a1e97","Type":"ContainerDied","Data":"461716f0a50510ebdae2a83a47aa5f2912a499e8ac8f13f1e50b29fcd44d6cd0"}
Oct 30 22:15:21.552351 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:21.552023    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461716f0a50510ebdae2a83a47aa5f2912a499e8ac8f13f1e50b29fcd44d6cd0"
Oct 30 22:15:21.613964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:21.613923    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"]
Oct 30 22:15:26.030382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.030340    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"]
Oct 30 22:15:26.041095 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.041064    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ad0ba-rknpl"]
Oct 30 22:15:26.533211 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.533172    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main" containerID="cri-o://71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54" gracePeriod=45
Oct 30 22:15:26.545560 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.545504    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" containerID="cri-o://e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659" gracePeriod=45
Oct 30 22:15:26.669496 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.669454    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused"
Oct 30 22:15:26.798218 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.798152    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 22:15:26.822680 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.822654    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 22:15:26.921728 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921687    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm\") pod \"5a7348fd-346f-4b77-a11d-2969029c5cf4\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") "
Oct 30 22:15:26.921916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921776    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache\") pod \"48a89f7d-9471-4be3-b234-010eb43156b7\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") "
Oct 30 22:15:26.921916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921807    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir\") pod \"48a89f7d-9471-4be3-b234-010eb43156b7\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") "
Oct 30 22:15:26.921916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921850    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir\") pod \"5a7348fd-346f-4b77-a11d-2969029c5cf4\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") "
Oct 30 22:15:26.921916 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921884    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh\" (UniqueName: \"kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh\") pod \"5a7348fd-346f-4b77-a11d-2969029c5cf4\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") "
Oct 30 22:15:26.922116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921923    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs\") pod \"5a7348fd-346f-4b77-a11d-2969029c5cf4\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") "
Oct 30 22:15:26.922116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.921967    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache\") pod \"5a7348fd-346f-4b77-a11d-2969029c5cf4\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") "
Oct 30 22:15:26.922116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.922037    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home\") pod \"5a7348fd-346f-4b77-a11d-2969029c5cf4\" (UID: \"5a7348fd-346f-4b77-a11d-2969029c5cf4\") "
Oct 30 22:15:26.922292 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.922253    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir" pod "48a89f7d-9471-4be3-b234-010eb43156b7" (UID: "48a89f7d-9471-4be3-b234-010eb43156b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.922382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.922323    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs\") pod \"48a89f7d-9471-4be3-b234-010eb43156b7\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") "
Oct 30 22:15:26.922382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.922372    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home\") pod \"48a89f7d-9471-4be3-b234-010eb43156b7\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") "
Oct 30 22:15:26.922506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.922468    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv\" (UniqueName: \"kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv\") pod \"48a89f7d-9471-4be3-b234-010eb43156b7\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") "
Oct 30 22:15:26.922581 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.922322    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache" pod "48a89f7d-9471-4be3-b234-010eb43156b7" (UID: "48a89f7d-9471-4be3-b234-010eb43156b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.923109 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.923069    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir" pod "5a7348fd-346f-4b77-a11d-2969029c5cf4" (UID: "5a7348fd-346f-4b77-a11d-2969029c5cf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.923341 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.923210    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home" pod "48a89f7d-9471-4be3-b234-010eb43156b7" (UID: "48a89f7d-9471-4be3-b234-010eb43156b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.923457 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.923430    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm\") pod \"48a89f7d-9471-4be3-b234-010eb43156b7\" (UID: \"48a89f7d-9471-4be3-b234-010eb43156b7\") "
Oct 30 22:15:26.923577 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.923552    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache" pod "5a7348fd-346f-4b77-a11d-2969029c5cf4" (UID: "5a7348fd-346f-4b77-a11d-2969029c5cf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.923659 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.923635    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home" pod "5a7348fd-346f-4b77-a11d-2969029c5cf4" (UID: "5a7348fd-346f-4b77-a11d-2969029c5cf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.924357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.924329    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.924480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.924360    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.924480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.924377    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.924480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.924392    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.924480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.924405    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.924480 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.924419    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.925488 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.925460    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm" pod "5a7348fd-346f-4b77-a11d-2969029c5cf4" (UID: "5a7348fd-346f-4b77-a11d-2969029c5cf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.925669 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.925614    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh" pod "5a7348fd-346f-4b77-a11d-2969029c5cf4" (UID: "5a7348fd-346f-4b77-a11d-2969029c5cf4"). InnerVolumeSpecName "kube-api-access-zbzrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:26.925766 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.925717    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs" pod "5a7348fd-346f-4b77-a11d-2969029c5cf4" (UID: "5a7348fd-346f-4b77-a11d-2969029c5cf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:26.925952 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.925927    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv" pod "48a89f7d-9471-4be3-b234-010eb43156b7" (UID: "48a89f7d-9471-4be3-b234-010eb43156b7"). InnerVolumeSpecName "kube-api-access-8xfqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:26.926251 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.926188    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm" pod "48a89f7d-9471-4be3-b234-010eb43156b7" (UID: "48a89f7d-9471-4be3-b234-010eb43156b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.926576 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:26.926554    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs" pod "48a89f7d-9471-4be3-b234-010eb43156b7" (UID: "48a89f7d-9471-4be3-b234-010eb43156b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:27.025215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.025177    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xfqv\" (UniqueName: \"kubernetes.io/projected/48a89f7d-9471-4be3-b234-010eb43156b7-kube-api-access-8xfqv\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.025215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.025209    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a89f7d-9471-4be3-b234-010eb43156b7-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.025215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.025219    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a7348fd-346f-4b77-a11d-2969029c5cf4-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.025475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.025227    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzrh\" (UniqueName: \"kubernetes.io/projected/5a7348fd-346f-4b77-a11d-2969029c5cf4-kube-api-access-zbzrh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.025475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.025237    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a7348fd-346f-4b77-a11d-2969029c5cf4-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.025475 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.025245    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a89f7d-9471-4be3-b234-010eb43156b7-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.213003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.212927    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23cab2c-9e7f-45ce-ae1b-263fce203269" path="/var/lib/kubelet/pods/c23cab2c-9e7f-45ce-ae1b-263fce203269/volumes"
Oct 30 22:15:27.267825 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.267792    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"]
Oct 30 22:15:27.609630 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.609574    6695 generic.go:356] "Generic (PLEG): container finished" podID="48a89f7d-9471-4be3-b234-010eb43156b7" containerID="71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54" exitCode=0
Oct 30 22:15:27.609797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.609638    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" event={"ID":"48a89f7d-9471-4be3-b234-010eb43156b7","Type":"ContainerDied","Data":"71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54"}
Oct 30 22:15:27.609797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.609653    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"
Oct 30 22:15:27.609797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.609684    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd" event={"ID":"48a89f7d-9471-4be3-b234-010eb43156b7","Type":"ContainerDied","Data":"e91fe22cab2b8e1e67624577427b3bc820fca5326aaeef0a62b721b861f3d143"}
Oct 30 22:15:27.609797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.609709    6695 scope.go:122] "RemoveContainer" containerID="71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54"
Oct 30 22:15:27.611345 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.611321    6695 generic.go:356] "Generic (PLEG): container finished" podID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerID="e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659" exitCode=0
Oct 30 22:15:27.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.611376    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"
Oct 30 22:15:27.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.611387    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" event={"ID":"5a7348fd-346f-4b77-a11d-2969029c5cf4","Type":"ContainerDied","Data":"e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659"}
Oct 30 22:15:27.611458 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.611419    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv" event={"ID":"5a7348fd-346f-4b77-a11d-2969029c5cf4","Type":"ContainerDied","Data":"89e8ddfe1a3a8081bfe7bd8d3e89d855cf45daf21147b64b5f8e33b22d63e282"}
Oct 30 22:15:27.620245 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.620227    6695 scope.go:122] "RemoveContainer" containerID="71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54"
Oct 30 22:15:27.620531 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:27.620511    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54\": container with ID starting with 71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54 not found: ID does not exist" containerID="71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54"
Oct 30 22:15:27.620587 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.620540    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54"} err="failed to get container status \"71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54\": rpc error: code = NotFound desc = could not find container \"71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54\": container with ID starting with 71ae79af6c7d2ac2a3378399f10f3658052bd3500d8eb51b6a63476bdbfcfb54 not found: ID does not exist"
Oct 30 22:15:27.620587 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.620557    6695 scope.go:122] "RemoveContainer" containerID="e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659"
Oct 30 22:15:27.629977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.629961    6695 scope.go:122] "RemoveContainer" containerID="e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659"
Oct 30 22:15:27.630250 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:27.630228    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659\": container with ID starting with e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659 not found: ID does not exist" containerID="e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659"
Oct 30 22:15:27.630328 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.630259    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659"} err="failed to get container status \"e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659\": rpc error: code = NotFound desc = could not find container \"e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659\": container with ID starting with e8880b9bb602095ad0f154dc7b3262f719912154b5b788a7c9dea0149e4c7659 not found: ID does not exist"
Oct 30 22:15:27.669744 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.669715    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"]
Oct 30 22:15:27.679491 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.679464    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/premium-w4-ea40ee-kserve-7566878f98-ghjqv"]
Oct 30 22:15:27.721755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.721728    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"]
Oct 30 22:15:27.726559 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:27.726536    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-2-w4-ea40ee-kserve-6985c76d74-x4nvd"]
Oct 30 22:15:28.035353 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035316    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"]
Oct 30 22:15:28.035761 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035747    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035762    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035775    6695 cpu_manager.go:414] "containerMap: removing container" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035781    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035795    6695 cpu_manager.go:414] "containerMap: removing container" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035800    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035806    6695 cpu_manager.go:414] "containerMap: removing container" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035811    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035818    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.035826 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035823    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035849    6695 cpu_manager.go:414] "containerMap: removing container" podUID="b34f18ae-421b-4a0b-b069-c174f51a1e97" containerName="revoke-keys"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035854    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34f18ae-421b-4a0b-b069-c174f51a1e97" containerName="revoke-keys"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035862    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035866    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035873    6695 cpu_manager.go:414] "containerMap: removing container" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035877    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035885    6695 cpu_manager.go:414] "containerMap: removing container" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035890    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035945    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34f18ae-421b-4a0b-b069-c174f51a1e97" containerName="revoke-keys"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035954    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035959    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035965    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035970    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" containerName="main"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035977    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035984    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" containerName="main"
Oct 30 22:15:28.036151 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.035989    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3f3fcc-9681-490b-9372-fd4e7f081385" containerName="cleanup"
Oct 30 22:15:28.038907 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.038889    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.057928 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.057900    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"]
Oct 30 22:15:28.058427 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.058408    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a30ad7-2868-41bc-becb-25b199488b2c" containerName="cleanup"
Oct 30 22:15:28.061665 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.061646    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.066694 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.066598    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:28.066830 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.066770    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:28.066900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.066834    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:28.066900 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.066884    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:15:28.067129 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.067095    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"]
Oct 30 22:15:28.103202 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.103168    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"]
Oct 30 22:15:28.139968 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.139930    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpv9\" (UniqueName: \"kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.140137 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.140079    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.140184 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.140138    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.241424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.241385    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.241424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.241426    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbsh\" (UniqueName: \"kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.241929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.241453    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpv9\" (UniqueName: \"kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.241929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.241761    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.241929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.241797    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.241929 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.241828    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.242428 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.242404    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.244114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.244094    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.261956 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.261925    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpv9\" (UniqueName: \"kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9\") pod \"maas-api-revoke-sub-3a29cd15-txsch\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.343640 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.343523    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.343810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.343700    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.343810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.343728    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbsh\" (UniqueName: \"kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.345403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.345366    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.348309 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.347173    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.353141 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.353113    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:28.363990 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.363968    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbsh\" (UniqueName: \"kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh\") pod \"maas-api-revoke-sub-9cecd249-szpfp\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.379923 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.379891    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:28.664724 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.664644    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"]
Oct 30 22:15:28.668208 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:15:28.668180    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2c64f5_0235_463e_b8bc_875b7b186f67.slice/crio-ec5f1babd5c3ea6a382d0789f948999e5db14552b5fba0b50978c8fccead5bb1 WatchSource:0}: Error finding container ec5f1babd5c3ea6a382d0789f948999e5db14552b5fba0b50978c8fccead5bb1: Status 404 returned error can't find the container with id ec5f1babd5c3ea6a382d0789f948999e5db14552b5fba0b50978c8fccead5bb1
Oct 30 22:15:28.703116 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.703088    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"]
Oct 30 22:15:28.707314 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:15:28.707285    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31824f80_519c_4d79_a05a_e1f6d239c0f4.slice/crio-883c5298c462f2359ac4f52444867c742aee14cc1de734b52af7dc86034515b9 WatchSource:0}: Error finding container 883c5298c462f2359ac4f52444867c742aee14cc1de734b52af7dc86034515b9: Status 404 returned error can't find the container with id 883c5298c462f2359ac4f52444867c742aee14cc1de734b52af7dc86034515b9
Oct 30 22:15:28.771575 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.771538    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"]
Oct 30 22:15:28.774855 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:28.774826    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ea40ee-285kc"]
Oct 30 22:15:29.216011 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.215975    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a89f7d-9471-4be3-b234-010eb43156b7" path="/var/lib/kubelet/pods/48a89f7d-9471-4be3-b234-010eb43156b7/volumes"
Oct 30 22:15:29.216657 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.216636    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7348fd-346f-4b77-a11d-2969029c5cf4" path="/var/lib/kubelet/pods/5a7348fd-346f-4b77-a11d-2969029c5cf4/volumes"
Oct 30 22:15:29.217074 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.217060    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ba3710-8177-43a3-99f6-b612a352c2df" path="/var/lib/kubelet/pods/d7ba3710-8177-43a3-99f6-b612a352c2df/volumes"
Oct 30 22:15:29.249448 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.249424    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 22:15:29.288410 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.288386    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 22:15:29.361764 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.361731    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume\") pod \"c52929c6-7e60-47f8-9861-1f277637b1c8\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") "
Oct 30 22:15:29.361933 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.361882    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r\" (UniqueName: \"kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r\") pod \"c52929c6-7e60-47f8-9861-1f277637b1c8\" (UID: \"c52929c6-7e60-47f8-9861-1f277637b1c8\") "
Oct 30 22:15:29.362209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.362179    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume" pod "c52929c6-7e60-47f8-9861-1f277637b1c8" (UID: "c52929c6-7e60-47f8-9861-1f277637b1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:29.363885 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.363863    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r" pod "c52929c6-7e60-47f8-9861-1f277637b1c8" (UID: "c52929c6-7e60-47f8-9861-1f277637b1c8"). InnerVolumeSpecName "kube-api-access-qvr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:29.462981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.462953    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb\" (UniqueName: \"kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb\") pod \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") "
Oct 30 22:15:29.463149 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.463104    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume\") pod \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\" (UID: \"0a8bded2-3c51-4e1d-a1b9-729171e6596c\") "
Oct 30 22:15:29.463416 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.463395    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvr6r\" (UniqueName: \"kubernetes.io/projected/c52929c6-7e60-47f8-9861-1f277637b1c8-kube-api-access-qvr6r\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:29.463470 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.463423    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52929c6-7e60-47f8-9861-1f277637b1c8-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:29.463518 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.463485    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume" pod "0a8bded2-3c51-4e1d-a1b9-729171e6596c" (UID: "0a8bded2-3c51-4e1d-a1b9-729171e6596c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:29.464887 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.464854    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb" pod "0a8bded2-3c51-4e1d-a1b9-729171e6596c" (UID: "0a8bded2-3c51-4e1d-a1b9-729171e6596c"). InnerVolumeSpecName "kube-api-access-gwghb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:29.564477 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.564449    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwghb\" (UniqueName: \"kubernetes.io/projected/0a8bded2-3c51-4e1d-a1b9-729171e6596c-kube-api-access-gwghb\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:29.564477 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.564475    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8bded2-3c51-4e1d-a1b9-729171e6596c-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:29.629265 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.629231    6695 generic.go:356] "Generic (PLEG): container finished" podID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerID="84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8" exitCode=0
Oct 30 22:15:29.629454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.629275    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"
Oct 30 22:15:29.629454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.629315    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" event={"ID":"c52929c6-7e60-47f8-9861-1f277637b1c8","Type":"ContainerDied","Data":"84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8"}
Oct 30 22:15:29.629454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.629381    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g" event={"ID":"c52929c6-7e60-47f8-9861-1f277637b1c8","Type":"ContainerDied","Data":"7ddcd3ea916ed7e52bdd7f6d559b9e8ec79e8a84871ff5a112337b0345422291"}
Oct 30 22:15:29.629454 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.629409    6695 scope.go:122] "RemoveContainer" containerID="84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8"
Oct 30 22:15:29.630843 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.630814    6695 generic.go:356] "Generic (PLEG): container finished" podID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerID="0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2" exitCode=0
Oct 30 22:15:29.630965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.630843    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" event={"ID":"0a8bded2-3c51-4e1d-a1b9-729171e6596c","Type":"ContainerDied","Data":"0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2"}
Oct 30 22:15:29.630965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.630864    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"
Oct 30 22:15:29.630965 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.630892    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb" event={"ID":"0a8bded2-3c51-4e1d-a1b9-729171e6596c","Type":"ContainerDied","Data":"cd141439d5e8b09adccd216b1187ba329e1ca61edff892e0cede1e74ff727271"}
Oct 30 22:15:29.632590 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.632567    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp" event={"ID":"31824f80-519c-4d79-a05a-e1f6d239c0f4","Type":"ContainerStarted","Data":"981371f7d5e27bda24dd510a76d188d0d91c54a83d5c4687790b16b4978e632a"}
Oct 30 22:15:29.632711 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.632624    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp" event={"ID":"31824f80-519c-4d79-a05a-e1f6d239c0f4","Type":"ContainerStarted","Data":"883c5298c462f2359ac4f52444867c742aee14cc1de734b52af7dc86034515b9"}
Oct 30 22:15:29.634556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.634530    6695 generic.go:356] "Generic (PLEG): container finished" podID="bd2c64f5-0235-463e-b8bc-875b7b186f67" containerID="938ff4a0d9a9d1e290b3eb0b2c3133c2d08c3d1c85f199fab29236831e90d00d" exitCode=0
Oct 30 22:15:29.634697 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.634680    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch" event={"ID":"bd2c64f5-0235-463e-b8bc-875b7b186f67","Type":"ContainerDied","Data":"938ff4a0d9a9d1e290b3eb0b2c3133c2d08c3d1c85f199fab29236831e90d00d"}
Oct 30 22:15:29.634758 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.634717    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch" event={"ID":"bd2c64f5-0235-463e-b8bc-875b7b186f67","Type":"ContainerStarted","Data":"ec5f1babd5c3ea6a382d0789f948999e5db14552b5fba0b50978c8fccead5bb1"}
Oct 30 22:15:29.640638 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.640513    6695 scope.go:122] "RemoveContainer" containerID="84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8"
Oct 30 22:15:29.640923 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:29.640897    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8\": container with ID starting with 84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8 not found: ID does not exist" containerID="84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8"
Oct 30 22:15:29.641026 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.640933    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8"} err="failed to get container status \"84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8\": rpc error: code = NotFound desc = could not find container \"84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8\": container with ID starting with 84847ca060621d7792a5a375acff47f07afd27086a9a11d4221f3ff54c9ecca8 not found: ID does not exist"
Oct 30 22:15:29.641026 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.640957    6695 scope.go:122] "RemoveContainer" containerID="0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2"
Oct 30 22:15:29.651755 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.651737    6695 scope.go:122] "RemoveContainer" containerID="0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2"
Oct 30 22:15:29.652071 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:29.652048    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2\": container with ID starting with 0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2 not found: ID does not exist" containerID="0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2"
Oct 30 22:15:29.652145 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.652077    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2"} err="failed to get container status \"0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2\": rpc error: code = NotFound desc = could not find container \"0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2\": container with ID starting with 0b7488f6627c476cb9464444454d21f2624f892417d2fb31b9c442eecd7308a2 not found: ID does not exist"
Oct 30 22:15:29.687732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.687652    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"]
Oct 30 22:15:29.693290 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.693259    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64d142-64cb68577f-gbjbb"]
Oct 30 22:15:29.731191 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.731167    6695 scope.go:122] "RemoveContainer" containerID="788a8cef47c3087d9f78931f0503573e37502b28ff75e63ba8a49589999b1992"
Oct 30 22:15:29.775449 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.775361    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp" podStartSLOduration=1.775343067 podStartE2EDuration="1.775343067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:29.773912531 +0000 UTC m=+3912.696370831" watchObservedRunningTime="2026-10-30 22:15:29.775343067 +0000 UTC m=+3912.697801365"
Oct 30 22:15:29.810239 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.810208    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"]
Oct 30 22:15:29.814858 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:29.814831    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64d142-656f4675bf-dg85g"]
Oct 30 22:15:30.642213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.642175    6695 generic.go:356] "Generic (PLEG): container finished" podID="31824f80-519c-4d79-a05a-e1f6d239c0f4" containerID="981371f7d5e27bda24dd510a76d188d0d91c54a83d5c4687790b16b4978e632a" exitCode=0
Oct 30 22:15:30.642690 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.642238    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp" event={"ID":"31824f80-519c-4d79-a05a-e1f6d239c0f4","Type":"ContainerDied","Data":"981371f7d5e27bda24dd510a76d188d0d91c54a83d5c4687790b16b4978e632a"}
Oct 30 22:15:30.781092 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.781065    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:30.878483 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.878444    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9\" (UniqueName: \"kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9\") pod \"bd2c64f5-0235-463e-b8bc-875b7b186f67\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") "
Oct 30 22:15:30.878696 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.878566    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca\") pod \"bd2c64f5-0235-463e-b8bc-875b7b186f67\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") "
Oct 30 22:15:30.878805 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.878707    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca\") pod \"bd2c64f5-0235-463e-b8bc-875b7b186f67\" (UID: \"bd2c64f5-0235-463e-b8bc-875b7b186f67\") "
Oct 30 22:15:30.879004 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.878967    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca" pod "bd2c64f5-0235-463e-b8bc-875b7b186f67" (UID: "bd2c64f5-0235-463e-b8bc-875b7b186f67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:30.880751 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.880728    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9" pod "bd2c64f5-0235-463e-b8bc-875b7b186f67" (UID: "bd2c64f5-0235-463e-b8bc-875b7b186f67"). InnerVolumeSpecName "kube-api-access-hhpv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:30.880838 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.880727    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca" pod "bd2c64f5-0235-463e-b8bc-875b7b186f67" (UID: "bd2c64f5-0235-463e-b8bc-875b7b186f67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:30.980403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.980318    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:30.980403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.980351    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpv9\" (UniqueName: \"kubernetes.io/projected/bd2c64f5-0235-463e-b8bc-875b7b186f67-kube-api-access-hhpv9\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:30.980403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:30.980362    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2c64f5-0235-463e-b8bc-875b7b186f67-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:31.073207 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:31.073171    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:15:31.212403 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.212367    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" path="/var/lib/kubelet/pods/0a8bded2-3c51-4e1d-a1b9-729171e6596c/volumes"
Oct 30 22:15:31.212786 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.212772    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" path="/var/lib/kubelet/pods/c52929c6-7e60-47f8-9861-1f277637b1c8/volumes"
Oct 30 22:15:31.657746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.657697    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch" event={"ID":"bd2c64f5-0235-463e-b8bc-875b7b186f67","Type":"ContainerDied","Data":"ec5f1babd5c3ea6a382d0789f948999e5db14552b5fba0b50978c8fccead5bb1"}
Oct 30 22:15:31.657746 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.657732    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a29cd15-txsch"
Oct 30 22:15:31.658284 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.657789    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5f1babd5c3ea6a382d0789f948999e5db14552b5fba0b50978c8fccead5bb1"
Oct 30 22:15:31.796560 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.796534    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:31.990313 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.990221    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh\" (UniqueName: \"kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh\") pod \"31824f80-519c-4d79-a05a-e1f6d239c0f4\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") "
Oct 30 22:15:31.990313 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.990271    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca\") pod \"31824f80-519c-4d79-a05a-e1f6d239c0f4\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") "
Oct 30 22:15:31.990586 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.990326    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca\") pod \"31824f80-519c-4d79-a05a-e1f6d239c0f4\" (UID: \"31824f80-519c-4d79-a05a-e1f6d239c0f4\") "
Oct 30 22:15:31.990792 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.990769    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca" pod "31824f80-519c-4d79-a05a-e1f6d239c0f4" (UID: "31824f80-519c-4d79-a05a-e1f6d239c0f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:31.992368 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.992347    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca" pod "31824f80-519c-4d79-a05a-e1f6d239c0f4" (UID: "31824f80-519c-4d79-a05a-e1f6d239c0f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:31.992487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:31.992463    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh" pod "31824f80-519c-4d79-a05a-e1f6d239c0f4" (UID: "31824f80-519c-4d79-a05a-e1f6d239c0f4"). InnerVolumeSpecName "kube-api-access-9fbsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:32.091903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.091865    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbsh\" (UniqueName: \"kubernetes.io/projected/31824f80-519c-4d79-a05a-e1f6d239c0f4-kube-api-access-9fbsh\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:32.091903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.091898    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-xks-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:32.091903 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.091911    6695 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31824f80-519c-4d79-a05a-e1f6d239c0f4-maas-api-openshift-ca\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:32.268487 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.268402    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerName="payload-pre-processing" containerID="cri-o://b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f" gracePeriod=25
Oct 30 22:15:32.666556 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.666523    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp"
Oct 30 22:15:32.667077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.666564    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cecd249-szpfp" event={"ID":"31824f80-519c-4d79-a05a-e1f6d239c0f4","Type":"ContainerDied","Data":"883c5298c462f2359ac4f52444867c742aee14cc1de734b52af7dc86034515b9"}
Oct 30 22:15:32.667077 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:32.666636    6695 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="883c5298c462f2359ac4f52444867c742aee14cc1de734b52af7dc86034515b9"
Oct 30 22:15:36.636551 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.636512    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" containerID="cri-o://a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78" gracePeriod=45
Oct 30 22:15:36.722922 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.722888    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"]
Oct 30 22:15:36.723112 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.723088    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy" containerID="cri-o://820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67" gracePeriod=30
Oct 30 22:15:36.892759 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.892699    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 22:15:36.943859 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.943827    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs\" (UniqueName: \"kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs\") pod \"98b7b70a-6cbf-43fb-94ec-7108fa483027\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") "
Oct 30 22:15:36.944040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.943882    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir\") pod \"98b7b70a-6cbf-43fb-94ec-7108fa483027\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") "
Oct 30 22:15:36.944040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.943923    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm\") pod \"98b7b70a-6cbf-43fb-94ec-7108fa483027\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") "
Oct 30 22:15:36.944040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.943953    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6\" (UniqueName: \"kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6\") pod \"98b7b70a-6cbf-43fb-94ec-7108fa483027\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") "
Oct 30 22:15:36.944040 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.944026    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home\") pod \"98b7b70a-6cbf-43fb-94ec-7108fa483027\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") "
Oct 30 22:15:36.944288 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.944142    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache\") pod \"98b7b70a-6cbf-43fb-94ec-7108fa483027\" (UID: \"98b7b70a-6cbf-43fb-94ec-7108fa483027\") "
Oct 30 22:15:36.944719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.944579    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home" pod "98b7b70a-6cbf-43fb-94ec-7108fa483027" (UID: "98b7b70a-6cbf-43fb-94ec-7108fa483027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.944719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.944657    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir" pod "98b7b70a-6cbf-43fb-94ec-7108fa483027" (UID: "98b7b70a-6cbf-43fb-94ec-7108fa483027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.944719 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.944686    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache" pod "98b7b70a-6cbf-43fb-94ec-7108fa483027" (UID: "98b7b70a-6cbf-43fb-94ec-7108fa483027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.948564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.947298    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm" pod "98b7b70a-6cbf-43fb-94ec-7108fa483027" (UID: "98b7b70a-6cbf-43fb-94ec-7108fa483027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.948564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.947853    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs" pod "98b7b70a-6cbf-43fb-94ec-7108fa483027" (UID: "98b7b70a-6cbf-43fb-94ec-7108fa483027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:36.950649 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:36.950586    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6" pod "98b7b70a-6cbf-43fb-94ec-7108fa483027" (UID: "98b7b70a-6cbf-43fb-94ec-7108fa483027"). InnerVolumeSpecName "kube-api-access-vr9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:37.045118 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.045084    6695 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b7b70a-6cbf-43fb-94ec-7108fa483027-tls-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.045118 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.045113    6695 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-tmp-dir\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.045118 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.045122    6695 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-dshm\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.045352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.045131    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9c6\" (UniqueName: \"kubernetes.io/projected/98b7b70a-6cbf-43fb-94ec-7108fa483027-kube-api-access-vr9c6\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.045352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.045140    6695 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-home\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.045352 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.045148    6695 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b7b70a-6cbf-43fb-94ec-7108fa483027-model-cache\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.068964 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.068931    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"]
Oct 30 22:15:37.074271 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.074243    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7061b7d6-75k7r"]
Oct 30 22:15:37.220744 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.220668    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6c1fab-882b-4f6d-aae9-72d5d96e7ec8" path="/var/lib/kubelet/pods/da6c1fab-882b-4f6d-aae9-72d5d96e7ec8/volumes"
Oct 30 22:15:37.701941 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.701900    6695 generic.go:356] "Generic (PLEG): container finished" podID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerID="a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78" exitCode=0
Oct 30 22:15:37.702377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.701961    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"
Oct 30 22:15:37.702377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.701963    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" event={"ID":"98b7b70a-6cbf-43fb-94ec-7108fa483027","Type":"ContainerDied","Data":"a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78"}
Oct 30 22:15:37.702377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.702066    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" event={"ID":"98b7b70a-6cbf-43fb-94ec-7108fa483027","Type":"ContainerDied","Data":"449fd0b0572b4876e52c753183790793b4d53f54cf8f07516a2f6813f3428153"}
Oct 30 22:15:37.702377 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.702082    6695 scope.go:122] "RemoveContainer" containerID="a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78"
Oct 30 22:15:37.712334 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.712313    6695 scope.go:122] "RemoveContainer" containerID="a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78"
Oct 30 22:15:37.712639 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:37.712615    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78\": container with ID starting with a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78 not found: ID does not exist" containerID="a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78"
Oct 30 22:15:37.712740 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.712651    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78"} err="failed to get container status \"a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78\": rpc error: code = NotFound desc = could not find container \"a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78\": container with ID starting with a43e1e8b2e8ad9fe092d70c6fa7ac8634b5e1e45be09b710f147d7c45a673a78 not found: ID does not exist"
Oct 30 22:15:37.758059 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.758028    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"]
Oct 30 22:15:37.771629 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.771588    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k"]
Oct 30 22:15:37.859674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:37.859633    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/simulator-w2-9d677d-kserve-c7d7885d-nhd9k" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:15:39.212159 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:39.212121    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" path="/var/lib/kubelet/pods/98b7b70a-6cbf-43fb-94ec-7108fa483027/volumes"
Oct 30 22:15:39.442428 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:39.442387    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:41.969117 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.969089    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 22:15:41.994100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994029    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994100 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994081    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994147    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994189    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994264    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994332    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:41.994465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994336    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994383    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:41.994465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994429    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994465 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994461    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994533    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf\") pod \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\" (UID: \"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae\") "
Oct 30 22:15:41.994797 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994567    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:41.994919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994804    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:41.994919 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994822    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:41.995038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994929    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:41.995038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994951    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:41.995038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994965    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:41.995038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994980    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:41.995038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.994994    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:41.996524 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.996499    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:41.996524 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.996509    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:41.996824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.996804    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "kube-api-access-tr5tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:41.996936 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:41.996864    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token" pod "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" (UID: "aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:42.096069 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.096034    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.096069 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.096065    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5tf\" (UniqueName: \"kubernetes.io/projected/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-kube-api-access-tr5tf\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.096069 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.096078    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.096329 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.096088    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.739281 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.739240    6695 generic.go:356] "Generic (PLEG): container finished" podID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerID="820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67" exitCode=0
Oct 30 22:15:42.739463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.739287    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"
Oct 30 22:15:42.739463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.739293    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" event={"ID":"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae","Type":"ContainerDied","Data":"820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67"}
Oct 30 22:15:42.739463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.739340    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n" event={"ID":"aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae","Type":"ContainerDied","Data":"1cb38db111ca7c5c40e6a9d368d5b483e6f5f76c4f11a04195adbd7d251f5255"}
Oct 30 22:15:42.739463 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.739364    6695 scope.go:122] "RemoveContainer" containerID="820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67"
Oct 30 22:15:42.749976 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.749956    6695 scope.go:122] "RemoveContainer" containerID="820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67"
Oct 30 22:15:42.750235 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:42.750219    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67\": container with ID starting with 820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67 not found: ID does not exist" containerID="820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67"
Oct 30 22:15:42.750294 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.750243    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67"} err="failed to get container status \"820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67\": rpc error: code = NotFound desc = could not find container \"820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67\": container with ID starting with 820b137f30fec7f3645fb3b60c65c3eb59b5f6ed1668e3e5bd9569fcf9866b67 not found: ID does not exist"
Oct 30 22:15:42.814364 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.814328    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"]
Oct 30 22:15:42.827950 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:42.827910    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-ttc6n"]
Oct 30 22:15:43.211981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:43.211937    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" path="/var/lib/kubelet/pods/aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae/volumes"
Oct 30 22:15:47.953979 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:47.953945    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"]
Oct 30 22:15:47.954429 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:47.954151    6695 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy" containerID="cri-o://615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb" gracePeriod=30
Oct 30 22:15:49.078048 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:49.078016    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c956c655b-c9565_ee8f671f-c470-4b14-8873-861278d4ea59/authorino/0.log"
Oct 30 22:15:49.442748 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:49.442654    6695 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:51.917730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:51.917694    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:15:52.539203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.539143    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 22:15:52.598789 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.598754    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume\") pod \"3f4f8cde-572b-4d6d-9977-1713faff4872\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") "
Oct 30 22:15:52.598981 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.598878    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl\" (UniqueName: \"kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl\") pod \"3f4f8cde-572b-4d6d-9977-1713faff4872\" (UID: \"3f4f8cde-572b-4d6d-9977-1713faff4872\") "
Oct 30 22:15:52.599215 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.599172    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume" pod "3f4f8cde-572b-4d6d-9977-1713faff4872" (UID: "3f4f8cde-572b-4d6d-9977-1713faff4872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:52.601317 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.601294    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl" pod "3f4f8cde-572b-4d6d-9977-1713faff4872" (UID: "3f4f8cde-572b-4d6d-9977-1713faff4872"). InnerVolumeSpecName "kube-api-access-4sfjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:52.687809 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.687770    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:15:52.700489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.700448    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfjl\" (UniqueName: \"kubernetes.io/projected/3f4f8cde-572b-4d6d-9977-1713faff4872-kube-api-access-4sfjl\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:52.700489 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.700487    6695 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f4f8cde-572b-4d6d-9977-1713faff4872-plugins-config-volume\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:52.802798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.802720    6695 generic.go:356] "Generic (PLEG): container finished" podID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerID="b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f" exitCode=0
Oct 30 22:15:52.802798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.802769    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"
Oct 30 22:15:52.803015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.802799    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" event={"ID":"3f4f8cde-572b-4d6d-9977-1713faff4872","Type":"ContainerDied","Data":"b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f"}
Oct 30 22:15:52.803015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.802880    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs" event={"ID":"3f4f8cde-572b-4d6d-9977-1713faff4872","Type":"ContainerDied","Data":"07387c78673aaf5355d53651ac7b9d9481cc95f58e386afec48bae7ed7699d29"}
Oct 30 22:15:52.803015 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.802905    6695 scope.go:122] "RemoveContainer" containerID="b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f"
Oct 30 22:15:52.813037 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.813018    6695 scope.go:122] "RemoveContainer" containerID="b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f"
Oct 30 22:15:52.813313 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:52.813291    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f\": container with ID starting with b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f not found: ID does not exist" containerID="b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f"
Oct 30 22:15:52.813401 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.813324    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f"} err="failed to get container status \"b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f\": rpc error: code = NotFound desc = could not find container \"b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f\": container with ID starting with b9eaeaafaed8773c4713a8e24e4284aa4d575eb9ff15a06ba79dde1edb406c3f not found: ID does not exist"
Oct 30 22:15:52.862389 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.862349    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"]
Oct 30 22:15:52.876685 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:52.876653    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9d677d-8f77498cf-vvfgs"]
Oct 30 22:15:53.185905 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.185878    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 22:15:53.211924 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.211895    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" path="/var/lib/kubelet/pods/3f4f8cde-572b-4d6d-9977-1713faff4872/volumes"
Oct 30 22:15:53.305774 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305744    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.305954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305792    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.305954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305814    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.305954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305887    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.305954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305920    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.305954 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305939    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.306267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305965    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.306267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.305984    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.306267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306023    6695 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket\") pod \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\" (UID: \"66df6bb2-e479-47cf-bbc1-9e8ba3202637\") "
Oct 30 22:15:53.306267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306095    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:53.306513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306282    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:53.306513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306314    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:53.306513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306462    6695 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-credential-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.306513 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306476    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:53.306798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306557    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:53.306798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306481    6695 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-certs\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.306798 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.306645    6695 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-data\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.308236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.308211    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:53.308236 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.308225    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "kube-api-access-dq5ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:53.308394 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.308231    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:53.308564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.308544    6695 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy" pod "66df6bb2-e479-47cf-bbc1-9e8ba3202637" (UID: "66df6bb2-e479-47cf-bbc1-9e8ba3202637"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:53.343706 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.343630    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:15:53.407510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.407474    6695 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5ph\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-kube-api-access-dq5ph\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.407510 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.407507    6695 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-token\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.407757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.407522    6695 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istiod-ca-cert\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.407757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.407534    6695 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-workload-socket\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.407757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.407545    6695 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-envoy\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.407757 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.407556    6695 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6bb2-e479-47cf-bbc1-9e8ba3202637-istio-podinfo\") on node \"ip-10-0-142-232.ec2.internal\" DevicePath \"\""
Oct 30 22:15:53.811455 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.811421    6695 generic.go:356] "Generic (PLEG): container finished" podID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerID="615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb" exitCode=0
Oct 30 22:15:53.811666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.811475    6695 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"
Oct 30 22:15:53.811666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.811506    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" event={"ID":"66df6bb2-e479-47cf-bbc1-9e8ba3202637","Type":"ContainerDied","Data":"615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb"}
Oct 30 22:15:53.811666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.811556    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59" event={"ID":"66df6bb2-e479-47cf-bbc1-9e8ba3202637","Type":"ContainerDied","Data":"bdc80895337536369bc334ade3776a962208d7c4f4381832d2b0c4c692d18688"}
Oct 30 22:15:53.811666 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.811613    6695 scope.go:122] "RemoveContainer" containerID="615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb"
Oct 30 22:15:53.822222 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.822203    6695 scope.go:122] "RemoveContainer" containerID="615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb"
Oct 30 22:15:53.822529 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:15:53.822503    6695 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb\": container with ID starting with 615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb not found: ID does not exist" containerID="615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb"
Oct 30 22:15:53.822633 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.822541    6695 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb"} err="failed to get container status \"615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb\": rpc error: code = NotFound desc = could not find container \"615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb\": container with ID starting with 615073e323ec93fa7ebc4c92c1c80b18fb18d995e11f6f12d1fa8008ebc46acb not found: ID does not exist"
Oct 30 22:15:53.868824 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.868793    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"]
Oct 30 22:15:53.882520 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:53.882493    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-6dx59"]
Oct 30 22:15:54.063501 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:54.063412    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c956c655b-c9565_ee8f671f-c470-4b14-8873-861278d4ea59/authorino/0.log"
Oct 30 22:15:55.211897 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:15:55.211861    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" path="/var/lib/kubelet/pods/66df6bb2-e479-47cf-bbc1-9e8ba3202637/volumes"
Oct 30 22:16:01.074349 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:16:01.074311    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:16:05.053810 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:05.053777    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:05.747010 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:05.746975    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:06.359138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:06.359104    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:07.343827 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:07.343792    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2dc05653-cqf2m_c12cca9f-b84d-4019-9f79-8866f7ac5c1c/revoke-keys/0.log"
Oct 30 22:16:07.536001 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:07.535961    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3970c8af-x2chp_717813ec-d5ff-4cac-9cfe-39a4aedf14b1/revoke-keys/0.log"
Oct 30 22:16:07.706882 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:07.706806    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3a29cd15-txsch_bd2c64f5-0235-463e-b8bc-875b7b186f67/revoke-keys/0.log"
Oct 30 22:16:07.895598 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:07.895568    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4922e3af-v7rgd_b34f18ae-421b-4a0b-b069-c174f51a1e97/revoke-keys/0.log"
Oct 30 22:16:08.087303 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:08.087269    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7e031d2f-z7sbl_29f1e7a3-a725-44b8-a54d-51bb006ab201/revoke-keys/0.log"
Oct 30 22:16:08.279152 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:08.279119    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7e17c031-g5d9p_65d91286-bcb2-465a-b179-1a6c5c8c4d30/revoke-keys/0.log"
Oct 30 22:16:08.468579 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:08.468500    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9cecd249-szpfp_31824f80-519c-4d79-a05a-e1f6d239c0f4/revoke-keys/0.log"
Oct 30 22:16:08.846291 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:08.846260    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cd2654d1-mccsn_fb053358-2e7c-4a86-b643-77308fa2200f/revoke-keys/0.log"
Oct 30 22:16:09.037347 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:09.037312    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d13eb1d2-t8gv9_5d0cfa60-a3fd-47c6-9b8b-532f5be42f8c/revoke-keys/0.log"
Oct 30 22:16:10.761859 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:10.761831    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c956c655b-c9565_ee8f671f-c470-4b14-8873-861278d4ea59/authorino/0.log"
Oct 30 22:16:13.143441 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:13.143411    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-d2jcl_4c5059d5-a52a-4288-abb9-a05c96f5f2dc/discovery/0.log"
Oct 30 22:16:13.564635 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:13.564575    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-pg25b_f315049a-d615-414b-a693-5401ebb4a8f7/kube-auth-proxy/0.log"
Oct 30 22:16:13.985790 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:13.985713    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-k86d4_ec504004-13c4-4d32-882e-a60f4b4d1db5/istio-proxy/0.log"
Oct 30 22:16:16.779481 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:16.779450    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hmvr9_583d659a-c61c-4378-a025-f370c0dd1472/main/0.log"
Oct 30 22:16:16.970850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:16.970823    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8wgfd7_b58e7aa0-bbfc-4718-bbe1-cf91a02e2785/main/0.log"
Oct 30 22:16:17.379705 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:17.379674    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gss94_064ef5f5-f2c4-43d8-ad8a-fcbf0b3d8cf7/main/0.log"
Oct 30 22:16:28.336839 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.336803    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vvnlf/must-gather-j8hdj"]
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337168    6695 cpu_manager.go:414] "containerMap: removing container" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerName="payload-pre-processing"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337179    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerName="payload-pre-processing"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337197    6695 cpu_manager.go:414] "containerMap: removing container" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337202    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337218    6695 cpu_manager.go:414] "containerMap: removing container" podUID="bd2c64f5-0235-463e-b8bc-875b7b186f67" containerName="revoke-keys"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337223    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2c64f5-0235-463e-b8bc-875b7b186f67" containerName="revoke-keys"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337231    6695 cpu_manager.go:414] "containerMap: removing container" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337235    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337244    6695 cpu_manager.go:414] "containerMap: removing container" podUID="31824f80-519c-4d79-a05a-e1f6d239c0f4" containerName="revoke-keys"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337248    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31824f80-519c-4d79-a05a-e1f6d239c0f4" containerName="revoke-keys"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337259    6695 cpu_manager.go:414] "containerMap: removing container" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerName="payload-processing"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337263    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerName="payload-processing"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337273    6695 cpu_manager.go:414] "containerMap: removing container" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337278    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337285    6695 cpu_manager.go:414] "containerMap: removing container" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerName="payload-pre-processing"
Oct 30 22:16:28.337325 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337290    6695 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerName="payload-pre-processing"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337346    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecb1a6c-ecf4-42e3-bf15-1fc1a3fe87ae" containerName="istio-proxy"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337357    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2c64f5-0235-463e-b8bc-875b7b186f67" containerName="revoke-keys"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337362    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="66df6bb2-e479-47cf-bbc1-9e8ba3202637" containerName="istio-proxy"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337368    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="31824f80-519c-4d79-a05a-e1f6d239c0f4" containerName="revoke-keys"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337373    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52929c6-7e60-47f8-9861-1f277637b1c8" containerName="payload-processing"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337379    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b7b70a-6cbf-43fb-94ec-7108fa483027" containerName="main"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337383    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4f8cde-572b-4d6d-9977-1713faff4872" containerName="payload-pre-processing"
Oct 30 22:16:28.337920 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.337389    6695 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8bded2-3c51-4e1d-a1b9-729171e6596c" containerName="payload-pre-processing"
Oct 30 22:16:28.340028 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.340008    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.359250 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.359220    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vvnlf\"/\"kube-root-ca.crt\""
Oct 30 22:16:28.361257 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.361234    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vvnlf/must-gather-j8hdj"]
Oct 30 22:16:28.362438 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.362418    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vvnlf\"/\"openshift-service-ca.crt\""
Oct 30 22:16:28.373261 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.373239    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vvnlf\"/\"default-dockercfg-gfcdw\""
Oct 30 22:16:28.434349 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.434316    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/797d31e7-927e-4d12-8f65-ed1b3c77363e-must-gather-output\") pod \"must-gather-j8hdj\" (UID: \"797d31e7-927e-4d12-8f65-ed1b3c77363e\") " pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.434500 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.434369    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvcx\" (UniqueName: \"kubernetes.io/projected/797d31e7-927e-4d12-8f65-ed1b3c77363e-kube-api-access-hnvcx\") pod \"must-gather-j8hdj\" (UID: \"797d31e7-927e-4d12-8f65-ed1b3c77363e\") " pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.535796 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.535757    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/797d31e7-927e-4d12-8f65-ed1b3c77363e-must-gather-output\") pod \"must-gather-j8hdj\" (UID: \"797d31e7-927e-4d12-8f65-ed1b3c77363e\") " pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.535977 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.535824    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvcx\" (UniqueName: \"kubernetes.io/projected/797d31e7-927e-4d12-8f65-ed1b3c77363e-kube-api-access-hnvcx\") pod \"must-gather-j8hdj\" (UID: \"797d31e7-927e-4d12-8f65-ed1b3c77363e\") " pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.536114 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.536091    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/797d31e7-927e-4d12-8f65-ed1b3c77363e-must-gather-output\") pod \"must-gather-j8hdj\" (UID: \"797d31e7-927e-4d12-8f65-ed1b3c77363e\") " pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.557155 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.557126    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvcx\" (UniqueName: \"kubernetes.io/projected/797d31e7-927e-4d12-8f65-ed1b3c77363e-kube-api-access-hnvcx\") pod \"must-gather-j8hdj\" (UID: \"797d31e7-927e-4d12-8f65-ed1b3c77363e\") " pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.655246 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.655177    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vvnlf/must-gather-j8hdj"
Oct 30 22:16:28.953375 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.953349    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vvnlf/must-gather-j8hdj"]
Oct 30 22:16:28.956142 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:16:28.956116    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod797d31e7_927e_4d12_8f65_ed1b3c77363e.slice/crio-4545d9908ef556de3d5814651a6c6ad962709cbee4561d4113995a885bbb5660 WatchSource:0}: Error finding container 4545d9908ef556de3d5814651a6c6ad962709cbee4561d4113995a885bbb5660: Status 404 returned error can't find the container with id 4545d9908ef556de3d5814651a6c6ad962709cbee4561d4113995a885bbb5660
Oct 30 22:16:28.958357 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:28.958338    6695 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:16:29.023564 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:29.023530    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vvnlf/must-gather-j8hdj" event={"ID":"797d31e7-927e-4d12-8f65-ed1b3c77363e","Type":"ContainerStarted","Data":"4545d9908ef556de3d5814651a6c6ad962709cbee4561d4113995a885bbb5660"}
Oct 30 22:16:29.902017 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:29.901988    6695 scope.go:122] "RemoveContainer" containerID="cbdcfd0438b22a9957bb12221b194b4fe82f2fc4a5430ef0ee9f3b7c366bc008"
Oct 30 22:16:29.918086 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:29.918061    6695 scope.go:122] "RemoveContainer" containerID="2cade2b838a8d30d5b37aff06d2fe01e1207d47b9783d9ea7753bc46cd889484"
Oct 30 22:16:29.941730 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:29.941711    6695 scope.go:122] "RemoveContainer" containerID="d355a062a11510616447bd0325416b329c60fac141d5e5f309ca5bd80b3c6a3b"
Oct 30 22:16:31.074378 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:16:31.074337    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:16:32.733156 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:32.733125    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8w46l_3c61a330-f062-4cbf-9794-3f7973e3826d/global-pull-secret-syncer/0.log"
Oct 30 22:16:33.900138 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:33.900057    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tn775_c567c895-2164-4fdc-a4b4-4df9af394efc/konnectivity-agent/0.log"
Oct 30 22:16:34.057207 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:34.057168    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vvnlf/must-gather-j8hdj" event={"ID":"797d31e7-927e-4d12-8f65-ed1b3c77363e","Type":"ContainerStarted","Data":"27f86ec24eec311eab3d80d28f23252bbbf06d9b48db8bfd55c49c3a4bb3b3d9"}
Oct 30 22:16:34.057419 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:34.057218    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vvnlf/must-gather-j8hdj" event={"ID":"797d31e7-927e-4d12-8f65-ed1b3c77363e","Type":"ContainerStarted","Data":"1006a4192063806d22d226f54fc0b7124d93f32172621f197a48376475e1aad7"}
Oct 30 22:16:34.098350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:34.098291    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vvnlf/must-gather-j8hdj" podStartSLOduration=1.431091517 podStartE2EDuration="6.098272252s" totalImagesPullingTime="4.667180735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:34.097321488 +0000 UTC m=+3977.019779786" watchObservedRunningTime="2026-10-30 22:16:34.098272252 +0000 UTC m=+3977.020730550"
Oct 30 22:16:34.272350 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:34.272318    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-232.ec2.internal_d772f23118a807c4f6e4705c5121a0bf/haproxy/0.log"
Oct 30 22:16:35.593193 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:35.593141    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:35.618232 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:35.618204    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:37.406414 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:37.406378    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:37.440310 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:37.440178    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:39.797351 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:39.797313    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c956c655b-c9565_ee8f671f-c470-4b14-8873-861278d4ea59/authorino/0.log"
Oct 30 22:16:40.000546 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:40.000511    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:40.015048 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:40.015018    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:41.492435 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:41.492397    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:41.528994 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:41.528959    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:43.135478 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.135442    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:43.181890 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.181846    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:43.638404 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.638356    6695 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"]
Oct 30 22:16:43.642287 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.642265    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.666924 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.666896    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"]
Oct 30 22:16:43.678038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.678012    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fx2nl\"/\"openshift-service-ca.crt\""
Oct 30 22:16:43.678038 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.678020    6695 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fx2nl\"/\"kube-root-ca.crt\""
Oct 30 22:16:43.678295 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.678108    6695 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fx2nl\"/\"default-dockercfg-sh698\""
Oct 30 22:16:43.784125 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.784092    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmcn\" (UniqueName: \"kubernetes.io/projected/dfc342fd-19ae-4051-85fa-61e507c60279-kube-api-access-bcmcn\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.784344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.784135    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-podres\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.784344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.784170    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-lib-modules\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.784344 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.784332    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-proc\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.784554 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.784367    6695 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-sys\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885470    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-proc\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885506 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885506    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-sys\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885560    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmcn\" (UniqueName: \"kubernetes.io/projected/dfc342fd-19ae-4051-85fa-61e507c60279-kube-api-access-bcmcn\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885581    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-proc\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885626    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-podres\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885666    6695 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-lib-modules\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885676    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-podres\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885685    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-sys\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.885850 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.885753    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfc342fd-19ae-4051-85fa-61e507c60279-lib-modules\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.908411 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.908349    6695 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmcn\" (UniqueName: \"kubernetes.io/projected/dfc342fd-19ae-4051-85fa-61e507c60279-kube-api-access-bcmcn\") pod \"perf-node-gather-daemonset-bjmxg\" (UID: \"dfc342fd-19ae-4051-85fa-61e507c60279\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:43.954324 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:43.954294    6695 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:44.264573 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.264488    6695 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"]
Oct 30 22:16:44.268109 ip-10-0-142-232 kubenswrapper[6695]: W1030 22:16:44.268077    6695 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddfc342fd_19ae_4051_85fa_61e507c60279.slice/crio-a9e0110f69e0276b5b02e989d328d9a97fc9c59cce9605164a7369ac5f363bdd WatchSource:0}: Error finding container a9e0110f69e0276b5b02e989d328d9a97fc9c59cce9605164a7369ac5f363bdd: Status 404 returned error can't find the container with id a9e0110f69e0276b5b02e989d328d9a97fc9c59cce9605164a7369ac5f363bdd
Oct 30 22:16:44.318596 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.318571    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/alertmanager/0.log"
Oct 30 22:16:44.441831 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.441807    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/config-reloader/0.log"
Oct 30 22:16:44.557909 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.557881    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/kube-rbac-proxy-web/0.log"
Oct 30 22:16:44.671455 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.671430    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/kube-rbac-proxy/0.log"
Oct 30 22:16:44.785466 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.785437    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/kube-rbac-proxy-metric/0.log"
Oct 30 22:16:44.898989 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:44.898907    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/prom-label-proxy/0.log"
Oct 30 22:16:45.033562 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.033530    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2b190e5a-08c3-4cdf-b224-d96fa2023ae2/init-config-reloader/0.log"
Oct 30 22:16:45.140009 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.139972    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg" event={"ID":"dfc342fd-19ae-4051-85fa-61e507c60279","Type":"ContainerStarted","Data":"e753386437b1fa44c8382f96f35c54da8f899c988e6c9423f1e0d68a9c40c5ca"}
Oct 30 22:16:45.140213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.140030    6695 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg" event={"ID":"dfc342fd-19ae-4051-85fa-61e507c60279","Type":"ContainerStarted","Data":"a9e0110f69e0276b5b02e989d328d9a97fc9c59cce9605164a7369ac5f363bdd"}
Oct 30 22:16:45.140213 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.140165    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:45.185126 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.185025    6695 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg" podStartSLOduration=2.185008252 podStartE2EDuration="2.185008252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:45.184195037 +0000 UTC m=+3988.106653333" watchObservedRunningTime="2026-10-30 22:16:45.185008252 +0000 UTC m=+3988.107466550"
Oct 30 22:16:45.279732 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.279703    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ljnpg_0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/kube-state-metrics/0.log"
Oct 30 22:16:45.342873 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.342833    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:45.391937 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.391908    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ljnpg_0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/kube-rbac-proxy-main/0.log"
Oct 30 22:16:45.488650 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.488555    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:45.505939 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.505910    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ljnpg_0d9f385c-ce7e-42d9-a79c-2ef0976f44f5/kube-rbac-proxy-self/0.log"
Oct 30 22:16:45.627267 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.627239    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54db77678f-lbdmq_2eed9c68-fae8-40b3-9bee-499e396d96e5/metrics-server/0.log"
Oct 30 22:16:45.744033 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:45.743960    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4x9mh_34a0ac7b-1822-4157-9438-d76e6783177e/monitoring-plugin/0.log"
Oct 30 22:16:46.256166 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:46.256137    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8tkj4_8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/node-exporter/0.log"
Oct 30 22:16:46.367505 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:46.367478    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8tkj4_8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/kube-rbac-proxy/0.log"
Oct 30 22:16:46.479316 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:46.479287    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8tkj4_8846e2f3-9d46-4bf1-a71e-2f9f09e6d85e/init-textfile/0.log"
Oct 30 22:16:46.968424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:46.968387    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-86ht2_b3d13165-1255-4e98-b0c8-a84b78043c93/kube-rbac-proxy-main/0.log"
Oct 30 22:16:46.979159 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:46.979113    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:47.040209 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:47.040177    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zgm89_f3109498-853f-4884-a87c-5103affc6b25/manager/0.log"
Oct 30 22:16:47.092255 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:47.092226    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-86ht2_b3d13165-1255-4e98-b0c8-a84b78043c93/kube-rbac-proxy-self/0.log"
Oct 30 22:16:47.209522 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:47.209487    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-86ht2_b3d13165-1255-4e98-b0c8-a84b78043c93/openshift-state-metrics/0.log"
Oct 30 22:16:48.712382 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:48.712209    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:48.949991 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:48.949958    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85f94b8bfc-8cgkb_67790f00-7ce5-4f74-ae11-893cf8e217ad/thanos-query/0.log"
Oct 30 22:16:48.996678 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:48.996563    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b9484675-8227r_e7f1f212-3b3e-4475-9d1f-2069f3f7002e/manager/0.log"
Oct 30 22:16:49.064946 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:49.064915    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85f94b8bfc-8cgkb_67790f00-7ce5-4f74-ae11-893cf8e217ad/kube-rbac-proxy-web/0.log"
Oct 30 22:16:49.180311 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:49.180281    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85f94b8bfc-8cgkb_67790f00-7ce5-4f74-ae11-893cf8e217ad/kube-rbac-proxy/0.log"
Oct 30 22:16:49.301493 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:49.301458    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85f94b8bfc-8cgkb_67790f00-7ce5-4f74-ae11-893cf8e217ad/prom-label-proxy/0.log"
Oct 30 22:16:49.419854 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:49.419804    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85f94b8bfc-8cgkb_67790f00-7ce5-4f74-ae11-893cf8e217ad/kube-rbac-proxy-rules/0.log"
Oct 30 22:16:49.532424 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:49.532381    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85f94b8bfc-8cgkb_67790f00-7ce5-4f74-ae11-893cf8e217ad/kube-rbac-proxy-metrics/0.log"
Oct 30 22:16:50.793849 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:50.793814    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:50.824003 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:50.823975    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-z7b58_c755a4aa-1eff-462d-9df5-ef8365fe69ae/proxy/0.log"
Oct 30 22:16:51.165002 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:51.164933    6695 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-bjmxg"
Oct 30 22:16:53.040250 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:53.040208    6695 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"]
Oct 30 22:16:53.052988 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:53.052951    6695 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e17c031-g5d9p"]
Oct 30 22:16:53.187378 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:53.187344    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7945d9969f-4ptgk_c1cde09f-1065-44f4-896a-a2ec6c6adcb8/console/0.log"
Oct 30 22:16:53.214503 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:53.214465    6695 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d91286-bcb2-465a-b179-1a6c5c8c4d30" path="/var/lib/kubelet/pods/65d91286-bcb2-465a-b179-1a6c5c8c4d30/volumes"
Oct 30 22:16:56.490370 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:56.490334    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-d2jcl_4c5059d5-a52a-4288-abb9-a05c96f5f2dc/discovery/0.log"
Oct 30 22:16:56.587512 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:56.587482    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-pg25b_f315049a-d615-414b-a693-5401ebb4a8f7/kube-auth-proxy/0.log"
Oct 30 22:16:56.699828 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:56.699797    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-k86d4_ec504004-13c4-4d32-882e-a60f4b4d1db5/istio-proxy/0.log"
Oct 30 22:16:57.171349 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:57.171307    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-d2jcl_4c5059d5-a52a-4288-abb9-a05c96f5f2dc/discovery/0.log"
Oct 30 22:16:57.784655 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:57.784622    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zwsqg_2ca7df3d-5d59-436f-a381-a0224e48272f/dns/0.log"
Oct 30 22:16:57.848425 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:57.848371    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-pg25b_f315049a-d615-414b-a693-5401ebb4a8f7/kube-auth-proxy/0.log"
Oct 30 22:16:57.894385 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:57.894353    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zwsqg_2ca7df3d-5d59-436f-a381-a0224e48272f/kube-rbac-proxy/0.log"
Oct 30 22:16:57.961196 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:57.961156    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-k86d4_ec504004-13c4-4d32-882e-a60f4b4d1db5/istio-proxy/0.log"
Oct 30 22:16:58.007674 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:58.007642    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9bv86_93b394b4-d048-4139-a5eb-c32918e048d2/dns-node-resolver/0.log"
Oct 30 22:16:59.617622 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:16:59.617577    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8v7dh_67203193-932d-445f-b14d-457040dcee8e/node-ca/0.log"
Oct 30 22:17:00.995277 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:17:00.995245    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-d2jcl_4c5059d5-a52a-4288-abb9-a05c96f5f2dc/discovery/0.log"
Oct 30 22:17:01.075188 ip-10-0-142-232 kubenswrapper[6695]: E1030 22:17:01.075151    6695 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7tff6" containerName="extractor"
Oct 30 22:17:01.089835 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:17:01.089800    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-pg25b_f315049a-d615-414b-a693-5401ebb4a8f7/kube-auth-proxy/0.log"
Oct 30 22:17:01.209203 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:17:01.209163    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-k86d4_ec504004-13c4-4d32-882e-a60f4b4d1db5/istio-proxy/0.log"
Oct 30 22:17:01.238927 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:17:01.238893    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c956c655b-c9565_ee8f671f-c470-4b14-8873-861278d4ea59/authorino/0.log"
Oct 30 22:17:02.052326 ip-10-0-142-232 kubenswrapper[6695]: I1030 22:17:02.052283    6695 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c956c655b-c9565_ee8f671f-c470-4b14-8873-861278d4ea59/authorino/0.log"